{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3e9baa5d",
   "metadata": {},
   "source": [
    "# Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "08579655",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded regression.py\n"
     ]
    }
   ],
   "source": [
    "from os.path import basename, exists\n",
    "\n",
    "\n",
    "def download(url):\n",
    "    filename = basename(url)\n",
    "    if not exists(filename):\n",
    "        from urllib.request import urlretrieve\n",
    "\n",
    "        local, _ = urlretrieve(url, filename)\n",
    "        print(\"Downloaded \" + local)\n",
    "\n",
    "\n",
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/thinkstats2.py\")\n",
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/thinkplot.py\")\n",
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/regression.py\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f2dfc492",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import thinkstats2\n",
    "import thinkplot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88201a93",
   "metadata": {},
   "source": [
    "The linear least squares fit in the previous chapter is an example of\n",
    "**regression**, which is the more general problem of fitting any kind of\n",
    "model to any kind of data. This use of the term \"regression\" is a\n",
    "historical accident; it is only indirectly related to the original\n",
    "meaning of the word.\n",
    "\n",
    "The goal of regression analysis is to describe the relationship between\n",
    "one set of variables, called the **dependent variables**, and another\n",
    "set of variables, called independent or **explanatory variables**.\n",
    "\n",
    "In the previous chapter we used mother's age as an explanatory variable\n",
    "to predict birth weight as a dependent variable. When there is only one\n",
    "dependent and one explanatory variable, that's **simple regression**. In\n",
    "this chapter, we move on to **multiple regression**, with more than one\n",
    "explanatory variable. If there is more than one dependent variable,\n",
    "that's multivariate regression."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17fd9d5c",
   "metadata": {},
   "source": [
    "If the relationship between the dependent and explanatory variable is\n",
    "linear, that's **linear regression**. For example, if the dependent\n",
    "variable is $y$ and the explanatory variables are $x_1$ and $x_2$, we\n",
    "would write the following linear regression model:\n",
    "$$y = \\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 + \\eps$$ where $\\beta_0$ is\n",
    "the intercept, $\\beta_1$ is the parameter associated with $x_1$,\n",
    "$\\beta_2$ is the parameter associated with $x_2$, and $\\eps$ is the\n",
    "residual due to random variation or other unknown factors.\n",
    "\n",
    "Given a sequence of values for $y$ and sequences for $x_1$ and $x_2$, we\n",
    "can find the parameters, $\\beta_0$, $\\beta_1$, and $\\beta_2$, that\n",
    "minimize the sum of $\\eps^2$. This process is called **ordinary least\n",
    "squares**. The computation is similar to `thinkstats2.LeastSquare`, but\n",
    "generalized to deal with more than one explanatory variable. You can\n",
    "find the details at\n",
    "<https://en.wikipedia.org/wiki/Ordinary_least_squares>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a46dbacf",
   "metadata": {},
   "source": [
    "## StatsModels\n",
    "\n",
    "In the previous chapter I presented `thinkstats2.LeastSquares`, an\n",
    "implementation of simple linear regression intended to be easy to read.\n",
    "For multiple regression we'll switch to StatsModels, a Python package\n",
    "that provides several forms of regression and other analyses. If you are\n",
    "using Anaconda, you already have StatsModels; otherwise you might have\n",
    "to install it.\n",
    "\n",
    "As an example, I'll run the model from the previous chapter with\n",
    "StatsModels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d7f0275f",
   "metadata": {},
   "outputs": [],
   "source": [
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/nsfg.py\")\n",
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/first.py\")\n",
    "\n",
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/2002FemPreg.dct\")\n",
    "download(\n",
    "    \"https://github.com/AllenDowney/ThinkStats2/raw/master/code/2002FemPreg.dat.gz\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bf9e310e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import first\n",
    "\n",
    "live, firsts, others = first.MakeFrames()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9cc0b70e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.formula.api as smf\n",
    "\n",
    "formula = 'totalwgt_lb ~ agepreg'\n",
    "model = smf.ols(formula, data=live)\n",
    "results = model.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "940a840e",
   "metadata": {},
   "source": [
    "`statsmodels` provides two interfaces (APIs); the \"formula\" API uses\n",
    "strings to identify the dependent and explanatory variables. It uses a\n",
    "syntax called `patsy`; in this example, the `~` operator separates the\n",
    "dependent variable on the left from the explanatory variables on the\n",
    "right.\n",
    "\n",
    "`smf.ols` takes the formula string and the DataFrame, `live`, and\n",
    "returns an OLS object that represents the model. The name `ols` stands\n",
    "for \"ordinary least squares.\"\n",
    "\n",
    "The `fit` method fits the model to the data and returns a\n",
    "RegressionResults object that contains the results.\n",
    "\n",
    "The results are also available as attributes. `params` is a Series that\n",
    "maps from variable names to their parameters, so we can get the\n",
    "intercept and slope like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e710ef24",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6.830396973311051, 0.017453851471802638)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inter = results.params['Intercept']\n",
    "slope = results.params['agepreg']\n",
    "inter, slope"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1c3e040",
   "metadata": {},
   "source": [
    "The estimated parameters are 6.83 and 0.0175, the same as from\n",
    "`LeastSquares`.\n",
    "\n",
    "`pvalues` is a Series that maps from variable names to the associated\n",
    "p-values, so we can check whether the estimated slope is statistically\n",
    "significant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f19f93b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "slope_pvalue = results.pvalues['agepreg']\n",
    "slope_pvalue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "42d0778e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.004738115474710369"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.rsquared"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22791bc7",
   "metadata": {},
   "source": [
    "The p-value associated with `agepreg` is `5.7e-11`, which is less than\n",
    "$0.001$, as expected.\n",
    "\n",
    "`results.rsquared` contains $R^2$, which is $0.0047$. `results` also\n",
    "provides `f_pvalue`, which is the p-value associated with the model as a\n",
    "whole, similar to testing whether $R^2$ is statistically significant.\n",
    "\n",
    "And `results` provides `resid`, a sequence of residuals, and\n",
    "`fittedvalues`, a sequence of fitted values corresponding to `agepreg`.\n",
    "\n",
    "The results object provides `summary()`, which represents the results in\n",
    "a readable format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2e2c19be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>totalwgt_lb</td>   <th>  R-squared:         </th> <td>   0.005</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.005</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   43.02</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Mon, 17 Jun 2024</td> <th>  Prob (F-statistic):</th> <td>5.72e-11</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>09:45:19</td>     <th>  Log-Likelihood:    </th> <td> -15897.</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>  9038</td>      <th>  AIC:               </th> <td>3.180e+04</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>  9036</td>      <th>  BIC:               </th> <td>3.181e+04</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     1</td>      <th>                     </th>     <td> </td>    \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>    \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th> <td>    6.8304</td> <td>    0.068</td> <td>  100.470</td> <td> 0.000</td> <td>    6.697</td> <td>    6.964</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>agepreg</th>   <td>    0.0175</td> <td>    0.003</td> <td>    6.559</td> <td> 0.000</td> <td>    0.012</td> <td>    0.023</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>1024.052</td> <th>  Durbin-Watson:     </th> <td>   1.618</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th>  <td> 0.000</td>  <th>  Jarque-Bera (JB):  </th> <td>3081.833</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>           <td>-0.601</td>  <th>  Prob(JB):          </th> <td>    0.00</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>       <td> 5.596</td>  <th>  Cond. No.          </th> <td>    118.</td>\n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/latex": [
       "\\begin{center}\n",
       "\\begin{tabular}{lclc}\n",
       "\\toprule\n",
       "\\textbf{Dep. Variable:}    &   totalwgt\\_lb   & \\textbf{  R-squared:         } &     0.005   \\\\\n",
       "\\textbf{Model:}            &       OLS        & \\textbf{  Adj. R-squared:    } &     0.005   \\\\\n",
       "\\textbf{Method:}           &  Least Squares   & \\textbf{  F-statistic:       } &     43.02   \\\\\n",
       "\\textbf{Date:}             & Mon, 17 Jun 2024 & \\textbf{  Prob (F-statistic):} &  5.72e-11   \\\\\n",
       "\\textbf{Time:}             &     09:45:19     & \\textbf{  Log-Likelihood:    } &   -15897.   \\\\\n",
       "\\textbf{No. Observations:} &        9038      & \\textbf{  AIC:               } & 3.180e+04   \\\\\n",
       "\\textbf{Df Residuals:}     &        9036      & \\textbf{  BIC:               } & 3.181e+04   \\\\\n",
       "\\textbf{Df Model:}         &           1      & \\textbf{                     } &             \\\\\n",
       "\\textbf{Covariance Type:}  &    nonrobust     & \\textbf{                     } &             \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "                   & \\textbf{coef} & \\textbf{std err} & \\textbf{t} & \\textbf{P$> |$t$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\textbf{Intercept} &       6.8304  &        0.068     &   100.470  &         0.000        &        6.697    &        6.964     \\\\\n",
       "\\textbf{agepreg}   &       0.0175  &        0.003     &     6.559  &         0.000        &        0.012    &        0.023     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lclc}\n",
       "\\textbf{Omnibus:}       & 1024.052 & \\textbf{  Durbin-Watson:     } &    1.618  \\\\\n",
       "\\textbf{Prob(Omnibus):} &   0.000  & \\textbf{  Jarque-Bera (JB):  } & 3081.833  \\\\\n",
       "\\textbf{Skew:}          &  -0.601  & \\textbf{  Prob(JB):          } &     0.00  \\\\\n",
       "\\textbf{Kurtosis:}      &   5.596  & \\textbf{  Cond. No.          } &     118.  \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "%\\caption{OLS Regression Results}\n",
       "\\end{center}\n",
       "\n",
       "Notes: \\newline\n",
       " [1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:            totalwgt_lb   R-squared:                       0.005\n",
       "Model:                            OLS   Adj. R-squared:                  0.005\n",
       "Method:                 Least Squares   F-statistic:                     43.02\n",
       "Date:                Mon, 17 Jun 2024   Prob (F-statistic):           5.72e-11\n",
       "Time:                        09:45:19   Log-Likelihood:                -15897.\n",
       "No. Observations:                9038   AIC:                         3.180e+04\n",
       "Df Residuals:                    9036   BIC:                         3.181e+04\n",
       "Df Model:                           1                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "==============================================================================\n",
       "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "Intercept      6.8304      0.068    100.470      0.000       6.697       6.964\n",
       "agepreg        0.0175      0.003      6.559      0.000       0.012       0.023\n",
       "==============================================================================\n",
       "Omnibus:                     1024.052   Durbin-Watson:                   1.618\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):             3081.833\n",
       "Skew:                          -0.601   Prob(JB):                         0.00\n",
       "Kurtosis:                       5.596   Cond. No.                         118.\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "\"\"\""
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c694792b",
   "metadata": {},
   "source": [
    "But it prints a lot of information that is not relevant (yet), so I use\n",
    "a simpler function called `SummarizeResults`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "950ce10d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept   6.83   (0)\n",
      "agepreg   0.0175   (5.72e-11)\n",
      "R^2 0.004738\n",
      "Std(ys) 1.408\n",
      "Std(res) 1.405\n"
     ]
    }
   ],
   "source": [
    "from regression import SummarizeResults\n",
    "\n",
    "SummarizeResults(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58114b68",
   "metadata": {},
   "source": [
    "`Std(ys)` is the standard deviation of the dependent variable, which is\n",
    "the RMSE if you have to guess birth weights without the benefit of any\n",
    "explanatory variables. `Std(res)` is the standard deviation of the\n",
    "residuals, which is the RMSE if your guesses are informed by the\n",
    "mother's age. As we have already seen, knowing the mother's age provides\n",
    "no substantial improvement to the predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f53611b4",
   "metadata": {},
   "source": [
    "## Multiple regression\n",
    "\n",
    "In Section [\\[birth_weights\\]](#birth_weights){reference-type=\"ref\"\n",
    "reference=\"birth_weights\"} we saw that first babies tend to be lighter\n",
    "than others, and this effect is statistically significant. But it is a\n",
    "strange result because there is no obvious mechanism that would cause\n",
    "first babies to be lighter. So we might wonder whether this relationship\n",
    "is **spurious**.\n",
    "\n",
    "In fact, there is a possible explanation for this effect. We have seen\n",
    "that birth weight depends on mother's age, and we might expect that\n",
    "mothers of first babies are younger than others.\n",
    "\n",
    "With a few calculations we can check whether this explanation is\n",
    "plausible. Then we'll use multiple regression to investigate more\n",
    "carefully. First, let's see how big the difference in weight is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3e62d320",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.12476118453549034"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diff_weight = firsts.totalwgt_lb.mean() - others.totalwgt_lb.mean()\n",
    "diff_weight"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba94643e",
   "metadata": {},
   "source": [
    "First babies are 0.125 lbs lighter, or 2 ounces. And the difference in\n",
    "ages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e65be931",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-3.5864347661500275"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diff_age = firsts.agepreg.mean() - others.agepreg.mean()\n",
    "diff_age"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86311655",
   "metadata": {},
   "source": [
    "The mothers of first babies are 3.59 years younger. Running the linear\n",
    "model again, we get the change in birth weight as a function of age:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e5dfbe3c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.017453851471802638"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = smf.ols('totalwgt_lb ~ agepreg', data=live).fit()\n",
    "slope = results.params['agepreg']\n",
    "slope"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad61e908",
   "metadata": {},
   "source": [
    "The slope is 0.0175 pounds per year. If we multiply the slope by the\n",
    "difference in ages, we get the expected difference in birth weight for\n",
    "first babies and others, due to mother's age:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "5c87490d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0625970997216918"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slope * diff_age"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eee213c",
   "metadata": {},
   "source": [
    "The result is 0.063, just about half of the observed difference. So we\n",
    "conclude, tentatively, that the observed difference in birth weight can\n",
    "be partly explained by the difference in mother's age.\n",
    "\n",
    "Using multiple regression, we can explore these relationships more\n",
    "systematically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "33340364",
   "metadata": {},
   "outputs": [],
   "source": [
    "live['isfirst'] = live.birthord == 1\n",
    "formula = 'totalwgt_lb ~ isfirst'\n",
    "results = smf.ols(formula, data=live).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa2113bf",
   "metadata": {},
   "source": [
    "The first line creates a new column named `isfirst` that is True for\n",
    "first babies and false otherwise. Then we fit a model using `isfirst` as\n",
    "an explanatory variable.\n",
    "\n",
    "Here are the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "deef5d4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept   7.33   (0)\n",
      "isfirst[T.True]   -0.125   (2.55e-05)\n",
      "R^2 0.00196\n",
      "Std(ys) 1.408\n",
      "Std(res) 1.407\n"
     ]
    }
   ],
   "source": [
    "SummarizeResults(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "685135dd",
   "metadata": {},
   "source": [
    "Because `isfirst` is a boolean, `ols` treats it as a **categorical\n",
    "variable**, which means that the values fall into categories, like True\n",
    "and False, and should not be treated as numbers. The estimated parameter\n",
    "is the effect on birth weight when `isfirst` is true, so the result,\n",
    "-0.125 lbs, is the difference in birth weight between first babies and\n",
    "others.\n",
    "\n",
    "The slope and the intercept are statistically significant, which means\n",
    "that they were unlikely to occur by chance, but the the $R^2$ value for\n",
    "this model is small, which means that `isfirst` doesn't account for a\n",
    "substantial part of the variation in birth weight.\n",
    "\n",
    "The results are similar with `agepreg`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "94342965",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept   6.83   (0)\n",
      "agepreg   0.0175   (5.72e-11)\n",
      "R^2 0.004738\n",
      "Std(ys) 1.408\n",
      "Std(res) 1.405\n"
     ]
    }
   ],
   "source": [
    "formula = 'totalwgt_lb ~ agepreg'\n",
    "results = smf.ols(formula, data=live).fit()\n",
    "SummarizeResults(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8f8338b",
   "metadata": {},
   "source": [
    "Again, the parameters are statistically significant, but $R^2$ is low.\n",
    "\n",
    "These models confirm results we have already seen. But now we can fit a\n",
    "single model that includes both variables. With the formula\n",
    "`totalwgt_lb ~ isfirst + agepreg`, we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "84a9be8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept   6.91   (0)\n",
      "isfirst[T.True]   -0.0698   (0.0253)\n",
      "agepreg   0.0154   (3.93e-08)\n",
      "R^2 0.005289\n",
      "Std(ys) 1.408\n",
      "Std(res) 1.405\n"
     ]
    }
   ],
   "source": [
    "formula = 'totalwgt_lb ~ isfirst + agepreg'\n",
    "results = smf.ols(formula, data=live).fit()\n",
    "SummarizeResults(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fff711b6",
   "metadata": {},
   "source": [
    "In the combined model, the parameter for `isfirst` is smaller by about\n",
    "half, which means that part of the apparent effect of `isfirst` is\n",
    "actually accounted for by `agepreg`. And the p-value for `isfirst` is\n",
    "about 2.5%, which is on the border of statistical significance.\n",
    "\n",
    "$R^2$ for this model is a little higher, which indicates that the two\n",
    "variables together account for more variation in birth weight than\n",
    "either alone (but not by much)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa7fd6f7",
   "metadata": {},
   "source": [
    "## Nonlinear relationships\n",
    "\n",
    "Remembering that the contribution of `agepreg` might be nonlinear, we\n",
    "might consider adding a variable to capture more of this relationship.\n",
    "One option is to create a column, `agepreg2`, that contains the squares\n",
    "of the ages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "7fe0861a",
   "metadata": {},
   "outputs": [],
   "source": [
    "live['agepreg2'] = live.agepreg**2\n",
    "formula = 'totalwgt_lb ~ isfirst + agepreg + agepreg2'\n",
    "results = smf.ols(formula, data=live).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49cd90c0",
   "metadata": {},
   "source": [
    "Now by estimating parameters for `agepreg` and `agepreg2`, we are\n",
    "effectively fitting a parabola:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "e31ed85c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept   6.91   (0)\n",
      "isfirst[T.True]   -0.0698   (0.0253)\n",
      "agepreg   0.0154   (3.93e-08)\n",
      "R^2 0.005289\n",
      "Std(ys) 1.408\n",
      "Std(res) 1.405\n"
     ]
    }
   ],
   "source": [
    "SummarizeResults(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "938c9f8b",
   "metadata": {},
   "source": [
    "The parameter of `agepreg2` is negative, so the parabola curves\n",
    "downward, which is consistent with the shape of the lines in\n",
    "Figure [\\[linear2\\]](#linear2){reference-type=\"ref\"\n",
    "reference=\"linear2\"}.\n",
    "\n",
    "The quadratic model of `agepreg` accounts for more of the variability in\n",
    "birth weight; the parameter for `isfirst` is smaller in this model, and\n",
    "no longer statistically significant.\n",
    "\n",
    "Using computed variables like `agepreg2` is a common way to fit\n",
    "polynomials and other functions to data. This process is still\n",
    "considered linear regression, because the dependent variable is a linear\n",
    "function of the explanatory variables, regardless of whether some\n",
    "variables are nonlinear functions of others.\n",
    "\n",
    "The following table summarizes the results of these regressions:\n",
    "\n",
    " center\n",
    "                 isfirst        agepreg     agepreg2     $R^2$\n",
    "  --------- ----------------- ----------- ------------- --------\n",
    "  Model 1       -0.125 \\*         --           --        0.002\n",
    "  Model 2          --          0.0175 \\*       --        0.0047\n",
    "  Model 3    -0.0698 (0.025)   0.0154 \\*       --        0.0053\n",
    "  Model 4    -0.0504 (0.11)    0.112 \\*    -0.00185 \\*   0.0075\n",
    "\n",
    "\n",
    "The columns in this table are the explanatory variables and the\n",
    "coefficient of determination, $R^2$. Each entry is an estimated\n",
    "parameter and either a p-value in parentheses or an asterisk to indicate\n",
    "a p-value less that 0.001.\n",
    "\n",
    "We conclude that the apparent difference in birth weight is explained,\n",
    "at least in part, by the difference in mother's age. When we include\n",
    "mother's age in the model, the effect of `isfirst` gets smaller, and the\n",
    "remaining effect might be due to chance.\n",
    "\n",
    "In this example, mother's age acts as a **control variable**; including\n",
    "`agepreg` in the model \"controls for\" the difference in age between\n",
    "first-time mothers and others, making it possible to isolate the effect\n",
    "(if any) of `isfirst`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3e615b9",
   "metadata": {},
   "source": [
    "## Data mining\n",
    "\n",
    "So far we have used regression models for explanation; for example, in\n",
    "the previous section we discovered that an apparent difference in birth\n",
    "weight is actually due to a difference in mother's age. But the $R^2$\n",
    "values of those models is very low, which means that they have little\n",
    "predictive power. In this section we'll try to do better.\n",
    "\n",
    "Suppose one of your co-workers is expecting a baby and there is an\n",
    "office pool to guess the baby's birth weight (if you are not familiar\n",
    "with betting pools, see <https://en.wikipedia.org/wiki/Betting_pool>).\n",
    "\n",
    "Now suppose that you *really* want to win the pool. What could you do to\n",
    "improve your chances? Well, the NSFG dataset includes 244 variables\n",
    "about each pregnancy and another 3087 variables about each respondent.\n",
    "Maybe some of those variables have predictive power. To find out which\n",
    "ones are most useful, why not try them all?\n",
    "\n",
    "Testing the variables in the pregnancy table is easy, but in order to\n",
    "use the variables in the respondent table, we have to match up each\n",
    "pregnancy with a respondent. In theory we could iterate through the rows\n",
    "of the pregnancy table, use the `caseid` to find the corresponding\n",
    "respondent, and copy the values from the correspondent table into the\n",
    "pregnancy table. But that would be slow.\n",
    "\n",
    "A better option is to recognize this process as a **join** operation as\n",
    "defined in SQL and other relational database languages (see\n",
    "<https://en.wikipedia.org/wiki/Join_(SQL)>). Join is implemented as a\n",
    "DataFrame method, so we can perform the operation like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "8111dfde",
   "metadata": {},
   "outputs": [],
   "source": [
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/2002FemResp.dct\")\n",
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/2002FemResp.dat.gz\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "b351e6ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nsfg\n",
    "\n",
    "live = live[live.prglngth>30]\n",
    "resp = nsfg.ReadFemResp()\n",
    "resp.index = resp.caseid\n",
    "join = live.join(resp, on='caseid', rsuffix='_r')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54979cf1",
   "metadata": {},
   "source": [
    "The first line selects records for pregnancies longer than 30 weeks,\n",
    "assuming that the office pool is formed several weeks before the due\n",
    "date.\n",
    "\n",
    "The next line reads the respondent file. The result is a DataFrame with\n",
    "integer indices; in order to look up respondents efficiently, I replace\n",
    "`resp.index` with `resp.caseid`.\n",
    "\n",
    "The `join` method is invoked on `live`, which is considered the \"left\"\n",
    "table, and passed `resp`, which is the \"right\" table. The keyword\n",
    "argument `on` indicates the variable used to match up rows from the two\n",
    "tables.\n",
    "\n",
    "In this example some column names appear in both tables, so we have to\n",
    "provide `rsuffix`, which is a string that will be appended to the names\n",
    "of overlapping columns from the right table. For example, both tables\n",
    "have a column named `race` that encodes the race of the respondent. The\n",
    "result of the join contains two columns named `race` and `race_r`.\n",
    "\n",
    "The Pandas implementation is fast. Joining the NSFG tables takes less\n",
    "than a second on an ordinary desktop computer. Now we can start testing\n",
    "variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "956efb90",
   "metadata": {},
   "outputs": [],
   "source": [
    "t = []\n",
    "for name in join.columns:\n",
    "    try:\n",
    "        if join[name].var() < 1e-7:\n",
    "            continue\n",
    "\n",
    "        formula = 'totalwgt_lb ~ agepreg + ' + name\n",
    "        model = smf.ols(formula, data=join)\n",
    "        if model.nobs < len(join)/2:\n",
    "            continue\n",
    "\n",
    "        results = model.fit()\n",
    "    except (ValueError, TypeError):\n",
    "        continue\n",
    "\n",
    "    t.append((results.rsquared, name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "5b31aea4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>$R^2$</th>\n",
       "      <th>column</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>totalwgt_lb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.949813</td>\n",
       "      <td>birthwgt_lb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>0.300824</td>\n",
       "      <td>lbw1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>0.130125</td>\n",
       "      <td>prglngth</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.123400</td>\n",
       "      <td>wksgest</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>0.102031</td>\n",
       "      <td>agecon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.027144</td>\n",
       "      <td>mosgest</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.018551</td>\n",
       "      <td>babysex</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>0.016200</td>\n",
       "      <td>race</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517</th>\n",
       "      <td>0.016200</td>\n",
       "      <td>race_r</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        $R^2$       column\n",
       "107  1.000000  totalwgt_lb\n",
       "12   0.949813  birthwgt_lb\n",
       "49   0.300824         lbw1\n",
       "39   0.130125     prglngth\n",
       "8    0.123400      wksgest\n",
       "44   0.102031       agecon\n",
       "9    0.027144      mosgest\n",
       "11   0.018551      babysex\n",
       "62   0.016200         race\n",
       "517  0.016200       race_r"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = ['$R^2$', 'column']\n",
    "pd.DataFrame(t, columns=columns).sort_values(by='$R^2$', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f4c3ce4",
   "metadata": {},
   "source": [
    "For each variable we construct a model, compute $R^2$, and append the\n",
    "results to a list. The models all include `agepreg`, since we already\n",
    "know that it has some predictive power.\n",
    "\n",
    "I check that each explanatory variable has some variability; otherwise\n",
    "the results of the regression are unreliable. I also check the number of\n",
    "observations for each model. Variables that contain a large number of\n",
    "`nan`s are not good candidates for prediction.\n",
    "\n",
    "For most of these variables, we haven't done any cleaning. Some of them\n",
    "are encoded in ways that don't work very well for linear regression. As\n",
    "a result, we might overlook some variables that would be useful if they\n",
    "were cleaned properly. But maybe we will find some good candidates."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77fb18b7",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "\n",
    "The next step is to sort the results and select the variables that yield\n",
    "the highest values of $R^2$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "5751b1cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def ReadVariables():\n",
    "    \"\"\"Reads Stata dictionary files for NSFG data.\n",
    "\n",
    "    returns: DataFrame that maps variables names to descriptions\n",
    "    \"\"\"\n",
    "    vars1 = thinkstats2.ReadStataDct('2002FemPreg.dct').variables\n",
    "    vars2 = thinkstats2.ReadStataDct('2002FemResp.dct').variables\n",
    "\n",
    "    all_vars = pd.concat([vars1, vars2])\n",
    "    all_vars.index = all_vars.name\n",
    "    return all_vars\n",
    "\n",
    "def MiningReport(variables, n=30):\n",
    "    \"\"\"Prints variables with the highest R^2.\n",
    "\n",
    "    t: list of (R^2, variable name) pairs\n",
    "    n: number of pairs to print\n",
    "    \"\"\"\n",
    "    all_vars = ReadVariables()\n",
    "\n",
    "    variables.sort(reverse=True)\n",
    "    for r2, name in variables[:n]:\n",
    "        key = re.sub('_r$', '', name)\n",
    "        try:\n",
    "            desc = all_vars.loc[key].desc\n",
    "            if isinstance(desc, pd.Series):\n",
    "                desc = desc[0]\n",
    "            print(name, r2, desc)\n",
    "        except (KeyError, IndexError):\n",
    "            print(name, r2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "5d08068c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "totalwgt_lb 1.0\n",
      "birthwgt_lb 0.9498127305978009 BD-3 BIRTHWEIGHT IN POUNDS - 1ST BABY FROM THIS PREGNANCY\n",
      "lbw1 0.3008240784470769 LOW BIRTHWEIGHT - BABY 1\n",
      "prglngth 0.13012519488625085 DURATION OF COMPLETED PREGNANCY IN WEEKS\n",
      "wksgest 0.12340041363361076 GESTATIONAL LENGTH OF COMPLETED PREGNANCY (IN WEEKS)\n",
      "agecon 0.10203149928156052 AGE AT TIME OF CONCEPTION\n",
      "mosgest 0.02714427463957958 GESTATIONAL LENGTH OF COMPLETED PREGNANCY (IN MONTHS)\n",
      "babysex 0.018550925293942533 BD-2 SEX OF 1ST LIVEBORN BABY FROM THIS PREGNANCY\n",
      "race_r 0.016199503586253106 RACE\n",
      "race 0.016199503586253106 RACE\n",
      "nbrnaliv 0.016017752709788113 BC-2 NUMBER OF BABIES BORN ALIVE FROM THIS PREGNANCY\n",
      "paydu 0.014003795578114597 IB-10 CURRENT LIVING QUARTERS OWNED/RENTED, ETC\n",
      "rmarout03 0.01343006646571343 INFORMAL MARITAL STATUS WHEN PREGNANCY ENDED - 3RD\n",
      "birthwgt_oz 0.013102457615706498 BD-3 BIRTHWEIGHT IN OUNCES - 1ST BABY FROM THIS PREGNANCY\n",
      "anynurse 0.012529022541810653 BH-1 WHETHER R BREASTFED THIS CHILD AT ALL - 1ST FROM THIS PREG\n",
      "bfeedwks 0.012193688404495417 DURATION OF BREASTFEEDING IN WEEKS\n",
      "totincr 0.011870069031173158 TOTAL INCOME OF R'S FAMILY\n",
      "marout03 0.011807801994375033 FORMAL MARITAL STATUS WHEN PREGNANCY ENDED - 3RD\n",
      "marcon03 0.011752599354395321 FORMAL MARITAL STATUS WHEN PREGNANCY BEGAN - 3RD\n",
      "cebow 0.011437770919637269 NUMBER OF CHILDREN BORN OUT OF WEDLOCK\n",
      "rmarout01 0.011407737138640073 INFORMAL MARITAL STATUS WHEN PREGNANCY ENDED - 1ST\n",
      "rmarout6 0.011354138472805753 INFORMAL MARITAL STATUS AT PREGNANCY OUTCOME - 6 CATEGORIES\n",
      "marout01 0.011269357246806444 FORMAL MARITAL STATUS WHEN PREGNANCY ENDED - 1ST\n",
      "hisprace_r 0.01123834930203138 RACE AND HISPANIC ORIGIN\n",
      "hisprace 0.01123834930203138 RACE AND HISPANIC ORIGIN\n",
      "mar1diss 0.0109615635907514 MONTHS BTW/1ST MARRIAGE & DISSOLUTION (OR INTERVIEW)\n",
      "fmarcon5 0.0106049646842995 FORMAL MARITAL STATUS AT CONCEPTION - 5 CATEGORIES\n",
      "rmarout02 0.010546913206564978 INFORMAL MARITAL STATUS WHEN PREGNANCY ENDED - 2ND\n",
      "marcon02 0.01048140179553414 FORMAL MARITAL STATUS WHEN PREGNANCY BEGAN - 2ND\n",
      "fmarout5 0.010461691367377068 FORMAL MARITAL STATUS AT PREGNANCY OUTCOME\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_171984/255888651.py:29: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`\n",
      "  desc = desc[0]\n"
     ]
    }
   ],
   "source": [
    "MiningReport(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dcaaf4a",
   "metadata": {},
   "source": [
    "The first variable on the list is `totalwgt_lb`, followed by\n",
    "`birthwgt_lb`. Obviously, we can't use birth weight to predict birth\n",
    "weight.\n",
    "\n",
    "Similarly `prglngth` has useful predictive power, but for the office\n",
    "pool we assume pregnancy length (and the related variables) are not\n",
    "known yet.\n",
    "\n",
    "The first useful predictive variable is `babysex` which indicates\n",
    "whether the baby is male or female. In the NSFG dataset, boys are about\n",
    "0.3 lbs heavier. So, assuming that the sex of the baby is known, we can\n",
    "use it for prediction.\n",
    "\n",
    "Next is `race`, which indicates whether the respondent is white, black,\n",
    "or other. As an explanatory variable, race can be problematic. In\n",
    "datasets like the NSFG, race is correlated with many other variables,\n",
    "including income and other socioeconomic factors. In a regression model,\n",
    "race acts as a **proxy variable**, so apparent correlations with race\n",
    "are often caused, at least in part, by other factors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67b97744",
   "metadata": {},
   "source": [
    "The next variable on the list is `nbrnaliv`, which indicates whether the\n",
    "pregnancy yielded multiple births. Twins and triplets tend to be smaller\n",
    "than other babies, so if we know whether our hypothetical co-worker is\n",
    "expecting twins, that would help.\n",
    "\n",
    "Next on the list is `paydu`, which indicates whether the respondent owns\n",
    "her home. It is one of several income-related variables that turn out to\n",
    "be predictive. In datasets like the NSFG, income and wealth are\n",
    "correlated with just about everything. In this example, income is\n",
    "related to diet, health, health care, and other factors likely to affect\n",
    "birth weight.\n",
    "\n",
    "Some of the other variables on the list are things that would not be\n",
    "known until later, like `bfeedwks`, the number of weeks the baby was\n",
    "breast fed. We can't use these variables for prediction, but you might\n",
    "want to speculate on reasons `bfeedwks` might be correlated with birth\n",
    "weight."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a4de56a",
   "metadata": {},
   "source": [
    "Sometimes you start with a theory and use data to test it. Other times\n",
    "you start with data and go looking for possible theories. The second\n",
    "approach, which this section demonstrates, is called **data mining**. An\n",
    "advantage of data mining is that it can discover unexpected patterns. A\n",
    "hazard is that many of the patterns it discovers are either random or\n",
    "spurious.\n",
    "\n",
    "Having identified potential explanatory variables, I tested a few models\n",
    "and settled on this one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "c5d4fc32",
   "metadata": {},
   "outputs": [],
   "source": [
    "formula = ('totalwgt_lb ~ agepreg + C(race) + babysex==1 + '\n",
    "           'nbrnaliv>1 + paydu==1 + totincr')\n",
    "results = smf.ols(formula, data=join).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ed528d6",
   "metadata": {},
   "source": [
    "This formula uses some syntax we have not seen yet: `C(race)` tells the\n",
    "formula parser (Patsy) to treat race as a categorical variable, even\n",
    "though it is encoded numerically.\n",
    "\n",
    "The encoding for `babysex` is 1 for male, 2 for female; writing\n",
    "`babysex==1` converts it to boolean, True for male and false for female.\n",
    "\n",
    "Similarly `nbrnaliv>1` is True for multiple births and `paydu==1` is\n",
    "True for respondents who own their houses.\n",
    "\n",
    "`totincr` is encoded numerically from 1-14, with each increment\n",
    "representing about \\$5000 in annual income.\n",
    "\n",
    "So we can treat these values as numerical, expressed in units of \\$5000.\n",
    "\n",
    "Here are the results of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "3e76ab37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept   6.63   (0)\n",
      "C(race)[T.2]   0.357   (5.43e-29)\n",
      "C(race)[T.3]   0.266   (2.33e-07)\n",
      "babysex == 1[T.True]   0.295   (5.39e-29)\n",
      "nbrnaliv > 1[T.True]   -1.38   (5.1e-37)\n",
      "paydu == 1[T.True]   0.12   (0.000114)\n",
      "agepreg   0.00741   (0.0035)\n",
      "totincr   0.0122   (0.00188)\n",
      "R^2 0.05999\n",
      "Std(ys) 1.271\n",
      "Std(res) 1.232\n"
     ]
    }
   ],
   "source": [
    "SummarizeResults(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c51b33cf",
   "metadata": {},
   "source": [
    "The estimated parameters for race are larger than I expected, especially\n",
    "since we control for income. The encoding is 1 for black, 2 for white,\n",
    "and 3 for other. Babies of black mothers are lighter than babies of\n",
    "other races by 0.27--0.36 lbs.\n",
    "\n",
    "As we've already seen, boys are heavier by about 0.3 lbs; twins and\n",
    "other multiplets are lighter by 1.4 lbs.\n",
    "\n",
    "People who own their homes have heavier babies by about 0.12 lbs, even\n",
    "when we control for income. The parameter for mother's age is smaller\n",
    "than what we saw in\n",
    "Section [\\[multiple\\]](#multiple){reference-type=\"ref\"\n",
    "reference=\"multiple\"}, which suggests that some of the other variables\n",
    "are correlated with age, probably including `paydu` and `totincr`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8f9b307",
   "metadata": {},
   "source": [
    "All of these variables are statistically significant, some with very low\n",
    "p-values, but $R^2$ is only 0.06, still quite small. RMSE without using\n",
    "the model is 1.27 lbs; with the model it drops to 1.23. So your chance\n",
    "of winning the pool is not substantially improved. Sorry!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e3a27da",
   "metadata": {},
   "source": [
    "## Logistic regression\n",
    "\n",
    "In the previous examples, some of the explanatory variables were\n",
    "numerical and some categorical (including boolean). But the dependent\n",
    "variable was always numerical.\n",
    "\n",
    "Linear regression can be generalized to handle other kinds of dependent\n",
    "variables. If the dependent variable is boolean, the generalized model\n",
    "is called **logistic regression**. If the dependent variable is an\n",
    "integer count, it's called **Poisson regression**.\n",
    "\n",
    "As an example of logistic regression, let's consider a variation on the\n",
    "office pool scenario. Suppose a friend of yours is pregnant and you want\n",
    "to predict whether the baby is a boy or a girl. You could use data from\n",
    "the NSFG to find factors that affect the \"sex ratio\", which is\n",
    "conventionally defined to be the probability of having a boy.\n",
    "\n",
    "If you encode the dependent variable numerically, for example 0 for a\n",
    "girl and 1 for a boy, you could apply ordinary least squares, but there\n",
    "would be problems. The linear model might be something like this:\n",
    "$$y = \\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 + \\eps$$ Where $y$ is the\n",
    "dependent variable, and $x_1$ and $x_2$ are explanatory variables. Then\n",
    "we could find the parameters that minimize the residuals.\n",
    "\n",
    "The problem with this approach is that it produces predictions that are\n",
    "hard to interpret. Given estimated parameters and values for $x_1$ and\n",
    "$x_2$, the model might predict $y=0.5$, but the only meaningful values\n",
    "of $y$ are 0 and 1.\n",
    "\n",
    "It is tempting to interpret a result like that as a probability; for\n",
    "example, we might say that a respondent with particular values of $x_1$\n",
    "and $x_2$ has a 50% chance of having a boy. But it is also possible for\n",
    "this model to predict $y=1.1$ or $y=-0.1$, and those are not valid\n",
    "probabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "546b65f1",
   "metadata": {},
   "source": [
    "Logistic regression avoids this problem by expressing predictions in\n",
    "terms of **odds** rather than probabilities. If you are not familiar\n",
    "with odds, \"odds in favor\" of an event is the ratio of the probability\n",
    "it will occur to the probability that it will not.\n",
    "\n",
    "So if I think my team has a 75% chance of winning, I would say that the\n",
    "odds in their favor are three to one, because the chance of winning is\n",
    "three times the chance of losing.\n",
    "\n",
    "Odds and probabilities are different representations of the same\n",
    "information. Given a probability, you can compute the odds like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "24aeac58",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = 0.75\n",
    "o = p / (1 - p)\n",
    "o"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b974b95",
   "metadata": {},
   "source": [
    "Given odds in favor, you can convert to probability like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "d3feb5fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = o / (o + 1)\n",
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a60864fb",
   "metadata": {},
   "source": [
    "Logistic regression is based on the following model:\n",
    "$$\\log o = \\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 + \\epsilon$$ Where $o$ is the\n",
    "odds in favor of a particular outcome; in the example, $o$ would be the\n",
    "odds of having a boy.\n",
    "\n",
    "Suppose we have estimated the parameters $\\beta_0$, $\\beta_1$, and\n",
    "$\\beta_2$ (I'll explain how in a minute). And suppose we are given\n",
    "values for $x_1$ and $x_2$. We can compute the predicted value of\n",
    "$\\log o$, and then convert to a probability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "d0ef6360",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7500000000000001"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_o = np.log(o)\n",
    "o = np.exp(log_o)\n",
    "p = o / (o + 1)\n",
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe956fa2",
   "metadata": {},
   "source": [
    "So in the office pool scenario we could compute the predictive\n",
    "probability of having a boy. But how do we estimate the parameters?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ddd8097",
   "metadata": {},
   "source": [
    "## Estimating parameters\n",
    "\n",
    "Unlike linear regression, logistic regression does not have a closed\n",
    "form solution, so it is solved by guessing an initial solution and\n",
    "improving it iteratively.\n",
    "\n",
    "The usual goal is to find the maximum-likelihood estimate (MLE), which\n",
    "is the set of parameters that maximizes the likelihood of the data. For\n",
    "example, suppose we have the following data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "b8c30564",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = np.array([0, 1, 0, 1])\n",
    "x1 = np.array([0, 0, 0, 1])\n",
    "x2 = np.array([0, 1, 1, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "573c6bec",
   "metadata": {},
   "source": [
    "And we start with the initial guesses $\\beta_0=-1.5$, $\\beta_1=2.8$, and\n",
    "$\\beta_2=1.1$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "a1371c94",
   "metadata": {},
   "outputs": [],
   "source": [
    "beta = [-1.5, 2.8, 1.1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce45ddb7",
   "metadata": {},
   "source": [
    "Then for each row we can compute `log_o`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "17255ecc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1.5, -0.4, -0.4,  2.4])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_o = beta[0] + beta[1] * x1 + beta[2] * x2 \n",
    "log_o"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "595bb451",
   "metadata": {},
   "source": [
    "And convert from log odds to probabilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "75642e08",
   "metadata": {},
   "outputs": [],
   "source": [
    "o = np.exp(log_o)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "ed010cba",
   "metadata": {},
   "outputs": [],
   "source": [
    "p = o / (o+1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ef1a31",
   "metadata": {},
   "source": [
    "Notice that when `log_o` is greater than 0, `o` is greater than 1 and\n",
    "`p` is greater than 0.5.\n",
    "\n",
    "The likelihood of an outcome is `p` when `y==1` and `1-p` when `y==0`.\n",
    "For example, if we think the probability of a boy is 0.8 and the outcome\n",
    "is a boy, the likelihood is 0.8; if the outcome is a girl, the\n",
    "likelihood is 0.2. We can compute that like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "e7c7357f",
   "metadata": {},
   "outputs": [],
   "source": [
    "likes = y * p + (1-y) * (1-p)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cd83085",
   "metadata": {},
   "source": [
    "The overall likelihood of the data is the product of `likes`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "b05e93f1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1800933529673034"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "like = np.prod(likes)\n",
    "like"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "459ec91b",
   "metadata": {},
   "source": [
    "For these values of `beta`, the likelihood of the data is 0.18. The goal\n",
    "of logistic regression is to find parameters that maximize this\n",
    "likelihood. To do that, most statistics packages use an iterative solver\n",
    "like Newton's method (see\n",
    "<https://en.wikipedia.org/wiki/Logistic_regression#Model_fitting>)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "279304ea",
   "metadata": {},
   "source": [
    "## Implementation\n",
    "\n",
    "StatsModels provides an implementation of logistic regression called\n",
    "`logit`, named for the function that converts from probability to log\n",
    "odds. To demonstrate its use, I'll look for variables that affect the\n",
    "sex ratio.\n",
    "\n",
    "Again, I load the NSFG data and select pregnancies longer than 30 weeks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "785b5340",
   "metadata": {},
   "outputs": [],
   "source": [
    "import first\n",
    "\n",
    "live, firsts, others = first.MakeFrames()\n",
    "live = live[live.prglngth>30]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1861b45d",
   "metadata": {},
   "source": [
    "`logit` requires the dependent variable to be binary (rather than\n",
    "boolean), so I create a new column named `boy`, using `astype(int)` to\n",
    "convert to binary integers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "7f27f727",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8884, 245)"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "live['boy'] = (live.babysex==1).astype(int)\n",
    "live.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b4b72aa",
   "metadata": {},
   "source": [
    "Factors that have been found to affect sex ratio include parents' age,\n",
    "birth order, race, and social status. We can use logistic regression to\n",
    "see if these effects appear in the NSFG data. I'll start with the\n",
    "mother's age:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "f89cfc31",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.formula.api as smf\n",
    "\n",
    "model = smf.logit('boy ~ agepreg', data=df)\n",
    "results = model.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f1b1af6",
   "metadata": {},
   "source": [
    "`logit` takes the same arguments as `ols`, a formula in Patsy syntax and\n",
    "a DataFrame. The result is a Logit object that represents the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "097eada9",
   "metadata": {},
   "source": [
    "The result of `model.fit` is a BinaryResults object, which is similar to\n",
    "the RegressionResults object we got from `ols`. Here is a summary of the\n",
    "results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "c5b0d5d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept   -0.0301   (0.772)\n",
      "C(race)[T.2]   -0.0224   (0.66)\n",
      "C(race)[T.3]   -0.000457   (0.996)\n",
      "agepreg   -0.00267   (0.629)\n",
      "hpagelb   0.0047   (0.266)\n",
      "birthord   0.00501   (0.821)\n",
      "R^2 0.000144\n"
     ]
    }
   ],
   "source": [
    "SummarizeResults(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b0f5a06",
   "metadata": {},
   "source": [
    "The parameter of `agepreg` is positive, which suggests that older\n",
    "mothers are more likely to have boys, but the p-value is 0.783, which\n",
    "means that the apparent effect could easily be due to chance.\n",
    "\n",
    "The coefficient of determination, $R^2$, does not apply to logistic\n",
    "regression, but there are several alternatives that are used as \"pseudo\n",
    "$R^2$ values.\" These values can be useful for comparing models. For\n",
    "example, here's a model that includes several factors believed to be\n",
    "associated with sex ratio:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "399c5393",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692944\n",
      "         Iterations 3\n"
     ]
    }
   ],
   "source": [
    "formula = 'boy ~ agepreg + hpagelb + birthord + C(race)'\n",
    "model = smf.logit(formula, data=df)\n",
    "results = model.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8dfa627",
   "metadata": {},
   "source": [
    "Along with mother's age, this model includes father's age at birth\n",
    "(`hpagelb`), birth order (`birthord`), and race as a categorical\n",
    "variable. Here are the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "b29faac2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept   -0.0301   (0.772)\n",
      "C(race)[T.2]   -0.0224   (0.66)\n",
      "C(race)[T.3]   -0.000457   (0.996)\n",
      "agepreg   -0.00267   (0.629)\n",
      "hpagelb   0.0047   (0.266)\n",
      "birthord   0.00501   (0.821)\n",
      "R^2 0.000144\n"
     ]
    }
   ],
   "source": [
    "SummarizeResults(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c5116e1",
   "metadata": {},
   "source": [
    "None of the estimated parameters are statistically significant. The\n",
    "pseudo-$R^2$ value is a little higher, but that could be due to chance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5e69215",
   "metadata": {},
   "source": [
    "## Accuracy\n",
    "\n",
    "In the office pool scenario, we are most interested in the accuracy of\n",
    "the model: the number of successful predictions, compared with what we\n",
    "would expect by chance.\n",
    "\n",
    "In the NSFG data, there are more boys than girls, so the baseline\n",
    "strategy is to guess \"boy\" every time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ddb47d3",
   "metadata": {},
   "source": [
    "The model contains attributes called `endog` and `exog` that contain the\n",
    "**endogenous variable**, another name for the dependent variable, and\n",
    "the **exogenous variables**, another name for the explanatory variables.\n",
    "Since they are NumPy arrays, it is sometimes convenient to convert them\n",
    "to DataFrames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "f8210b20",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "endog = pd.DataFrame(model.endog, columns=[model.endog_names])\n",
    "exog = pd.DataFrame(model.exog, columns=model.exog_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ef8ce69",
   "metadata": {},
   "source": [
    "The accuracy of this strategy is\n",
    "just the fraction of boys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "80148060",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8782,)"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "actual = endog['boy']\n",
    "actual.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "7cb8dc3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.507173764518333"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "baseline = actual.mean()\n",
    "baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72d583cb",
   "metadata": {},
   "source": [
    "Since `actual` is encoded in binary integers, the mean is the fraction\n",
    "of boys, which is 0.507.\n",
    "\n",
    "Here's how we compute the accuracy of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "59dd28ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "predict = (results.predict() >= 0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "0b6b5343",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3944.0"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true_pos = predict * actual\n",
    "true_pos.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "a1d2e9a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "548.0"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true_neg = (1 - predict) * (1 - actual)\n",
    "true_neg.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22de46d8",
   "metadata": {},
   "source": [
    "`results.predict` returns a NumPy array of probabilities, which we round\n",
    "off to 0 or 1. Multiplying by `actual` yields 1 if we predict a boy and\n",
    "get it right, 0 otherwise. So, `true_pos` indicates \"true positives\".\n",
    "\n",
    "Similarly, `true_neg` indicates the cases where we guess \"girl\" and get\n",
    "it right. Accuracy is the fraction of correct guesses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "5ccd35a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5115007970849464"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc = (sum(true_pos) + sum(true_neg)) / len(actual)\n",
    "acc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6a5b8d0",
   "metadata": {},
   "source": [
    "The result is 0.512, slightly better than the baseline, 0.507. But, you\n",
    "should not take this result too seriously. We used the same data to\n",
    "build and test the model, so the model may not have predictive power on\n",
    "new data.\n",
    "\n",
    "Nevertheless, let's use the model to make a prediction for the office\n",
    "pool. Suppose your friend is 35 years old and white, her husband is 39,\n",
    "and they are expecting their third child:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "08f93a7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = ['agepreg', 'hpagelb', 'birthord', 'race']\n",
    "new = pd.DataFrame([[35, 39, 3, 2]], columns=columns)\n",
    "y = results.predict(new)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eca4ddc3",
   "metadata": {},
   "source": [
    "To invoke `results.predict` for a new case, you have to construct a\n",
    "DataFrame with a column for each variable in the model. The result in\n",
    "this case is 0.52, so you should guess \"boy.\" But if the model improves\n",
    "your chances of winning, the difference is very small."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0a3487c",
   "metadata": {},
   "source": [
    "## Glossary\n",
    "\n",
    "-   **regression**: One of several related processes for estimating\n",
    "    parameters that fit a model to data.\n",
    "\n",
    "-   **dependent variables**: The variables in a regression model we\n",
    "    would like to predict. Also known as endogenous variables.\n",
    "\n",
    "-   **explanatory variables**: The variables used to predict or explain\n",
    "    the dependent variables. Also known as independent, or exogenous,\n",
    "    variables.\n",
    "\n",
    "-   **simple regression**: A regression with only one dependent and one\n",
    "    explanatory variable.\n",
    "\n",
    "-   **multiple regression**: A regression with multiple explanatory\n",
    "    variables, but only one dependent variable.\n",
    "\n",
    "-   **linear regression**: A regression based on a linear model.\n",
    "\n",
    "-   **ordinary least squares**: A linear regression that estimates\n",
    "    parameters by minimizing the squared error of the residuals.\n",
    "\n",
    "-   **spurious relationship**: A relationship between two variables that\n",
    "    is caused by a statistical artifact or a factor, not included in the\n",
    "    model, that is related to both variables.\n",
    "\n",
    "-   **control variable**: A variable included in a regression to\n",
    "    eliminate or \"control for\" a spurious relationship.\n",
    "\n",
    "-   **proxy variable**: A variable that contributes information to a\n",
    "    regression model indirectly because of a relationship with another\n",
    "    factor, so it acts as a proxy for that factor.\n",
    "\n",
    "-   **categorical variable**: A variable that can have one of a discrete\n",
    "    set of unordered values.\n",
    "\n",
    "-   **join**: An operation that combines data from two DataFrames using\n",
    "    a key to match up rows in the two frames.\n",
    "\n",
    "-   **data mining**: An approach to finding relationships between\n",
    "    variables by testing a large number of models.\n",
    "\n",
    "-   **logistic regression**: A form of regression used when the\n",
    "    dependent variable is boolean.\n",
    "\n",
    "-   **Poisson regression**: A form of regression used when the dependent\n",
    "    variable is a non-negative integer, usually a count.\n",
    "\n",
    "-   **odds**: An alternative way of representing a probability, $p$, as\n",
    "    the ratio of the probability and its complement, $p / (1-p)$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e334137",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fe41625",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Exercise:** Suppose one of your co-workers is expecting a baby and you are participating in an office pool to predict the date of birth. Assuming that bets are placed during the 30th week of pregnancy, what variables could you use to make the best prediction? You should limit yourself to variables that are known before the birth, and likely to be available to the people in the pool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "36e5ea85",
   "metadata": {},
   "outputs": [],
   "source": [
    "import first\n",
    "live, firsts, others = first.MakeFrames()\n",
    "live = live[live.prglngth>30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "363434ec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>        <td>prglngth</td>     <th>  R-squared:         </th> <td>   0.011</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.011</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   34.28</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Mon, 17 Jun 2024</td> <th>  Prob (F-statistic):</th> <td>5.09e-22</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>10:22:06</td>     <th>  Log-Likelihood:    </th> <td> -18247.</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>  8884</td>      <th>  AIC:               </th> <td>3.650e+04</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>  8880</td>      <th>  BIC:               </th> <td>3.653e+04</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     3</td>      <th>                     </th>     <td> </td>    \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>    \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "            <td></td>               <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>             <td>   38.7617</td> <td>    0.039</td> <td> 1006.410</td> <td> 0.000</td> <td>   38.686</td> <td>   38.837</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>birthord == 1[T.True]</th> <td>    0.1015</td> <td>    0.040</td> <td>    2.528</td> <td> 0.011</td> <td>    0.023</td> <td>    0.180</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>race == 2[T.True]</th>     <td>    0.1390</td> <td>    0.042</td> <td>    3.311</td> <td> 0.001</td> <td>    0.057</td> <td>    0.221</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>nbrnaliv > 1[T.True]</th>  <td>   -1.4944</td> <td>    0.164</td> <td>   -9.086</td> <td> 0.000</td> <td>   -1.817</td> <td>   -1.172</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>1587.470</td> <th>  Durbin-Watson:     </th> <td>   1.619</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th>  <td> 0.000</td>  <th>  Jarque-Bera (JB):  </th> <td>6160.751</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>           <td>-0.852</td>  <th>  Prob(JB):          </th> <td>    0.00</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>       <td> 6.707</td>  <th>  Cond. No.          </th> <td>    10.9</td>\n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/latex": [
       "\\begin{center}\n",
       "\\begin{tabular}{lclc}\n",
       "\\toprule\n",
       "\\textbf{Dep. Variable:}         &     prglngth     & \\textbf{  R-squared:         } &     0.011   \\\\\n",
       "\\textbf{Model:}                 &       OLS        & \\textbf{  Adj. R-squared:    } &     0.011   \\\\\n",
       "\\textbf{Method:}                &  Least Squares   & \\textbf{  F-statistic:       } &     34.28   \\\\\n",
       "\\textbf{Date:}                  & Mon, 17 Jun 2024 & \\textbf{  Prob (F-statistic):} &  5.09e-22   \\\\\n",
       "\\textbf{Time:}                  &     10:22:06     & \\textbf{  Log-Likelihood:    } &   -18247.   \\\\\n",
       "\\textbf{No. Observations:}      &        8884      & \\textbf{  AIC:               } & 3.650e+04   \\\\\n",
       "\\textbf{Df Residuals:}          &        8880      & \\textbf{  BIC:               } & 3.653e+04   \\\\\n",
       "\\textbf{Df Model:}              &           3      & \\textbf{                     } &             \\\\\n",
       "\\textbf{Covariance Type:}       &    nonrobust     & \\textbf{                     } &             \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "                                & \\textbf{coef} & \\textbf{std err} & \\textbf{t} & \\textbf{P$> |$t$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\textbf{Intercept}              &      38.7617  &        0.039     &  1006.410  &         0.000        &       38.686    &       38.837     \\\\\n",
       "\\textbf{birthord == 1[T.True]}  &       0.1015  &        0.040     &     2.528  &         0.011        &        0.023    &        0.180     \\\\\n",
       "\\textbf{race == 2[T.True]}      &       0.1390  &        0.042     &     3.311  &         0.001        &        0.057    &        0.221     \\\\\n",
       "\\textbf{nbrnaliv $>$ 1[T.True]} &      -1.4944  &        0.164     &    -9.086  &         0.000        &       -1.817    &       -1.172     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lclc}\n",
       "\\textbf{Omnibus:}       & 1587.470 & \\textbf{  Durbin-Watson:     } &    1.619  \\\\\n",
       "\\textbf{Prob(Omnibus):} &   0.000  & \\textbf{  Jarque-Bera (JB):  } & 6160.751  \\\\\n",
       "\\textbf{Skew:}          &  -0.852  & \\textbf{  Prob(JB):          } &     0.00  \\\\\n",
       "\\textbf{Kurtosis:}      &   6.707  & \\textbf{  Cond. No.          } &     10.9  \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "%\\caption{OLS Regression Results}\n",
       "\\end{center}\n",
       "\n",
       "Notes: \\newline\n",
       " [1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:               prglngth   R-squared:                       0.011\n",
       "Model:                            OLS   Adj. R-squared:                  0.011\n",
       "Method:                 Least Squares   F-statistic:                     34.28\n",
       "Date:                Mon, 17 Jun 2024   Prob (F-statistic):           5.09e-22\n",
       "Time:                        10:22:06   Log-Likelihood:                -18247.\n",
       "No. Observations:                8884   AIC:                         3.650e+04\n",
       "Df Residuals:                    8880   BIC:                         3.653e+04\n",
       "Df Model:                           3                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "=========================================================================================\n",
       "                            coef    std err          t      P>|t|      [0.025      0.975]\n",
       "-----------------------------------------------------------------------------------------\n",
       "Intercept                38.7617      0.039   1006.410      0.000      38.686      38.837\n",
       "birthord == 1[T.True]     0.1015      0.040      2.528      0.011       0.023       0.180\n",
       "race == 2[T.True]         0.1390      0.042      3.311      0.001       0.057       0.221\n",
       "nbrnaliv > 1[T.True]     -1.4944      0.164     -9.086      0.000      -1.817      -1.172\n",
       "==============================================================================\n",
       "Omnibus:                     1587.470   Durbin-Watson:                   1.619\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):             6160.751\n",
       "Skew:                          -0.852   Prob(JB):                         0.00\n",
       "Kurtosis:                       6.707   Cond. No.                         10.9\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "\"\"\""
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "# The following are the only variables I found that have a statistically significant effect on pregnancy length.\n",
    "\n",
    "import statsmodels.formula.api as smf\n",
    "model = smf.ols('prglngth ~ birthord==1 + race==2 + nbrnaliv>1', data=live)\n",
    "results = model.fit()\n",
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea22658b",
   "metadata": {},
   "source": [
    "**Exercise:** The Trivers-Willard hypothesis suggests that for many mammals the sex ratio depends on “maternal condition”; that is, factors like the mother’s age, size, health, and social status. See https://en.wikipedia.org/wiki/Trivers-Willard_hypothesis\n",
    "\n",
    "Some studies have shown this effect among humans, but results are mixed. In this chapter we tested some variables related to these factors, but didn’t find any with a statistically significant effect on sex ratio.\n",
    "\n",
    "As an exercise, use a data mining approach to test the other variables in the pregnancy and respondent files. Can you find any factors with a substantial effect?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "ca0a00d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692991\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692961\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692849\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692996\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692903\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692724\n",
      "         Iterations 6\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692992\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692577\n",
      "         Iterations 6\n",
      "Warning: Maximum number of iterations has been exceeded.\n",
      "         Current function value: inf\n",
      "         Iterations: 35\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.686599\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693022\n",
      "         Iterations 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:2385: RuntimeWarning: overflow encountered in exp\n",
      "  return 1/(1+np.exp(-X))\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:2443: RuntimeWarning: divide by zero encountered in log\n",
      "  return np.sum(np.log(self.cdf(q * linpred)))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692903\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693031\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692963\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692964\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692905\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693092\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693107\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692589\n",
      "         Iterations 6\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692774\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692843\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692612\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692947\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692817\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692947\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693023\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692967\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692995\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692645\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692555\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693021\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693066\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692911\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692917\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692861\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692859\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692814\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692389\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692517\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692452\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692701\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692290\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692930\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692926\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693004\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692930\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692861\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692866\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692866\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692806\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692930\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692989\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692987\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692993\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692871\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692994\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693001\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692942\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692922\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692573\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692988\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692817\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692588\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692877\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692652\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692991\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692918\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692984\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692952\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692893\n",
      "         Iterations 5\n",
      "Warning: Maximum number of iterations has been exceeded.\n",
      "         Current function value: 0.692776\n",
      "         Iterations: 35\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692638\n",
      "         Iterations 6\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692838\n",
      "         Iterations 5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/base/model.py:607: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals\n",
      "  warnings.warn(\"Maximum Likelihood optimization failed to \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692971\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692971\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692810\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692995\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692979\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.686305\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692959\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692991\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692984\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692928\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692910\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693007\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692881\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692866\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692861\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692866\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692930\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692806\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692808\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693004\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693039\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692886\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692957\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692888\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692989\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692921\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692996\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692907\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692439\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692916\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692951\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692950\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693037\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693021\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693018\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692850\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693056\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692885\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693007\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692963\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692994\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692959\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692319\n",
      "         Iterations 6\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692984\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692841\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692879\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693007\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692871\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692993\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692997\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692956\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692780\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693055\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693104\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693022\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693069\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693052\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692988\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693037\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693063\n",
      "         Iterations 3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693052\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693078\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693078\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692964\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692801\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693074\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692959\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692995\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693004\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692911\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692833\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693025\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692823\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692970\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692945\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693055\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692908\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692703\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692853\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692981\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693053\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692991\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692962\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692928\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693009\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692999\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692943\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693049\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692901\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692951\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692843\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692834\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692783\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692799\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692850\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692841\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693118\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693128\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692958\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692898\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693005\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692829\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692999\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692972\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692901\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692929\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692894\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692820\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692960\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692999\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693007\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692993\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693002\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692968\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692995\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693028\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692992\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692733\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692645\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693029\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692842\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692821\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692721\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693054\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692742\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693007\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692958\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692975\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692848\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692942\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692953\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692915\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692931\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692940\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692992\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693000\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692999\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692979\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693000\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692984\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692978\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692930\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692829\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692992\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692980\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692971\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693000\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693000\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692979\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692996\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692990\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692950\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693037\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693039\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693091\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693065\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692992\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692987\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692840\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692984\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693002\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693023\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693001\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693016\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692970\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692980\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692989\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693007\n",
      "         Iterations 3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692994\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692990\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692965\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692994\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692818\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692980\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692680\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692674\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692884\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692865\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693007\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692883\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692994\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692941\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692948\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692917\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692993\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692999\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692968\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692850\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692971\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692844\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692963\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692980\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692984\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692937\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692960\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692970\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693001\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692911\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693002\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692929\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692995\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692845\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692981\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692718\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692575\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692673\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692988\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692778\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693018\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692804\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693023\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692752\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693023\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692671\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692709\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693046\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692650\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693063\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692875\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693098\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692806\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693074\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693058\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693024\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692892\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692959\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692853\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692615\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692979\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693046\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692917\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692977\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693004\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692932\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692619\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692779\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692886\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692739\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692662\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692621\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692792\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692926\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692879\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692805\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693009\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692673\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692940\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692903\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693005\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692994\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692992\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692980\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693001\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692999\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693002\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692941\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692972\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692952\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692977\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692920\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692856\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692955\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692852\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692573\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692976\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692879\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693069\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692955\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692868\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693018\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693027\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692949\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692945\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692962\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692971\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692579\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692949\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692801\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692987\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693094\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693036\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692917\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692971\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692824\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692862\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692862\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692976\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693009\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693009\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692965\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692967\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692916\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693005\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692951\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692962\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692952\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692946\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692995\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692974\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692992\n",
      "         Iterations 3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692995\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692994\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692866\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692806\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692989\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692987\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693005\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693007\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692610\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692904\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692859\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692712\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692948\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692993\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692871\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692883\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692801\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693001\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692994\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692722\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692909\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692561\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692803\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692870\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692755\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692702\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692963\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692879\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692855\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692837\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692729\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693001\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692990\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692875\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692989\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692813\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692475\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693001\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692833\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692529\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692898\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692640\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692784\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692875\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692782\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692938\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692871\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692939\n",
      "         Iterations 18\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693009\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693005\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692976\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692995\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692958\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693002\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692983\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692955\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692967\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693009\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692859\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692925\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692969\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692983\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692894\n",
      "         Iterations 4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692874\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692859\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692924\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692982\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692914\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692969\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692983\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692695\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692955\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692946\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693009\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692859\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692925\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692969\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692983\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692955\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692889\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692823\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692968\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692859\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692925\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692969\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692983\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692989\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692997\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692923\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692990\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692929\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693001\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693005\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692939\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692990\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692929\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692991\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692966\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692941\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692978\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692929\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692974\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692930\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692929\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692933\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693045\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693078\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692739\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692997\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692750\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692991\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692873\n",
      "         Iterations 3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692830\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693082\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692636\n",
      "         Iterations 3\n",
      "Warning: Maximum number of iterations has been exceeded.\n",
      "         Current function value: 0.692709\n",
      "         Iterations: 35\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692863\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692926\n",
      "         Iterations 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/base/model.py:607: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals\n",
      "  warnings.warn(\"Maximum Likelihood optimization failed to \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692726\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692774\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692999\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692861\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692705\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692723\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692803\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692956\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692786\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693007\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692989\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692993\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692952\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692970\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693009\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 4\n",
      "Warning: Maximum number of iterations has been exceeded.\n",
      "         Current function value: 0.692862\n",
      "         Iterations: 35\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692861\n",
      "         Iterations 4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/base/model.py:607: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals\n",
      "  warnings.warn(\"Maximum Likelihood optimization failed to \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692658\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692789\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692855\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692855\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692749\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692862\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692862\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692821\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692772\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692687\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692947\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692771\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692755\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693006\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692949\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692912\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692879\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692950\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692991\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692929\n",
      "         Iterations 4\n",
      "Warning: Maximum number of iterations has been exceeded.\n",
      "         Current function value: 0.692960\n",
      "         Iterations: 35\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693007\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692981\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692975\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692961\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693005\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692995\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693001\n",
      "         Iterations 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/base/model.py:607: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals\n",
      "  warnings.warn(\"Maximum Likelihood optimization failed to \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692957\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692983\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692258\n",
      "         Iterations 5\n",
      "Warning: Maximum number of iterations has been exceeded.\n",
      "         Current function value: 0.692696\n",
      "         Iterations: 35\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692993\n",
      "         Iterations 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/base/model.py:607: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals\n",
      "  warnings.warn(\"Maximum Likelihood optimization failed to \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693009\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692853\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692971\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692639\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692917\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692760\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692832\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693028\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692888\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692770\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692992\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692976\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692866\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692992\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692976\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692866\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692896\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692758\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693081\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692927\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692827\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692890\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692941\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692920\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692920\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692982\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693001\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692997\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693005\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692956\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692931\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692996\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692996\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692730\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692898\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692879\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692887\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692987\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692995\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692980\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692849\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692974\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692987\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692957\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693009\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693002\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693013\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692980\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692849\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692974\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692987\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692957\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693009\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693002\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692969\n",
      "         Iterations 4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693005\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692946\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692862\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692905\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692962\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692977\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693010\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693000\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692998\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693012\n",
      "         Iterations 3\n",
      "Warning: Maximum number of iterations has been exceeded.\n",
      "         Current function value: 0.692939\n",
      "         Iterations: 35\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692831\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692999\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/base/model.py:607: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals\n",
      "  warnings.warn(\"Maximum Likelihood optimization failed to \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692997\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692795\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692693\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692457\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692815\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693002\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692989\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693008\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692673\n",
      "         Iterations 5\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692982\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692985\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692986\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692922\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692973\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693011\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693015\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692942\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692921\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693002\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692810\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692995\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692979\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693014\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.693003\n",
      "         Iterations 3\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692996\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692996\n",
      "         Iterations 4\n",
      "Optimization terminated successfully.\n",
      "         Current function value: 0.692971\n",
      "         Iterations 3\n",
      "Warning: Maximum number of iterations has been exceeded.\n",
      "         Current function value: 0.000000\n",
      "         Iterations: 35\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/discrete/discrete_model.py:227: PerfectSeparationWarning: Perfect separation or prediction detected, parameter may not be identified\n",
      "  warnings.warn(msg, category=PerfectSeparationWarning)\n",
      "/home/downey/miniconda3/envs/ThinkStats/lib/python3.10/site-packages/statsmodels/base/model.py:607: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals\n",
      "  warnings.warn(\"Maximum Likelihood optimization failed to \"\n"
     ]
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "def GoMining(df):\n",
    "    \"\"\"Searches for variables that predict birth weight.\n",
    "\n",
    "    df: DataFrame of pregnancy records\n",
    "\n",
    "    returns: list of (rsquared, variable name) pairs\n",
    "    \"\"\"\n",
    "    df['boy'] = (df.babysex==1).astype(int)\n",
    "    variables = []\n",
    "    for name in df.columns:\n",
    "        try:\n",
    "            if df[name].var() < 1e-7:\n",
    "                continue\n",
    "\n",
    "            formula='boy ~ agepreg + ' + name\n",
    "            model = smf.logit(formula, data=df)\n",
    "            nobs = len(model.endog)\n",
    "            if nobs < len(df)/2:\n",
    "                continue\n",
    "\n",
    "            results = model.fit()\n",
    "        except:\n",
    "            continue\n",
    "\n",
    "        variables.append((results.prsquared, name))\n",
    "\n",
    "    return variables\n",
    "\n",
    "variables = GoMining(join)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "44f0eef3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "boy 0.9999999999839987\n",
      "totalwgt_lb 0.009696855253233383\n",
      "birthwgt_lb 0.009274460080281988 BD-3 BIRTHWEIGHT IN POUNDS - 1ST BABY FROM THIS PREGNANCY\n",
      "constat3 0.0010985419170438382 3RD PRIORITY CODE FOR CURRENT CONTRACEPTIVE STATUS\n",
      "lbw1 0.0010519527860078925 LOW BIRTHWEIGHT - BABY 1\n",
      "nplaced 0.001010368752280555 # OF R'S BIO CHILDREN SHE PLACED FOR ADOPTION (BASED ON BPA)\n",
      "fmarout5 0.0009096579032891183 FORMAL MARITAL STATUS AT PREGNANCY OUTCOME\n",
      "rmarout6 0.000818252143711895 INFORMAL MARITAL STATUS AT PREGNANCY OUTCOME - 6 CATEGORIES\n",
      "infever 0.0008115919859909004 EVER USED INFERTILITY SERVICES OF ANY KIND\n",
      "frsteatd 0.0007675331422082321 AGE (IN MOS) WHEN 1ST SUPPLEMENTED - 1ST FROM THIS PREG\n",
      "splstwk1 0.0007334122339932581 IF-1 H/P DOING WHAT LAST WEEK (EMPLOYMENT STATUS) 1ST MENTION\n",
      "pmarpreg 0.0007245809157658822 WHETHER PREGNANCY ENDED BEFORE R'S 1ST MARRIAGE (PREMARITALLY)\n",
      "usefstp 0.0007122387685902787 EF-3 USE METHOD AT FIRST SEX WITH 1ST PARTNER IN PAST 12 MONTHS?\n",
      "outcom02 0.0007015744602576479 OUTCOME OF PREGNANCY - 2ND\n",
      "nummult34 0.0006606172426639745 NUMBER OF METHODS REPORTED IN (OCT 2001)\n",
      "coh1dur 0.0006550110146368304 DURATION (IN MONTHS) OF R'S FIRST COHABITATION\n",
      "brnout_r 0.0006438582787924307 IB-8 R BORN OUTSIDE OF US\n",
      "brnout 0.0006438582787924307 IB-8 R BORN OUTSIDE OF US\n",
      "bpa_bdscheck1 0.0006384992730465999 WHETHER 1ST LIVEBORN BABY FROM THIS PREGNANCY WAS BPA OR BDS\n",
      "nummult41 0.000624852395994635 NUMBER OF METHODS REPORTED IN (MAY 2002)\n",
      "agepreg_i 0.0006221097878729154 AGEPREG IMPUTATION FLAG\n",
      "educmom 0.0005903653895802385 MOTHER'S (OR MOTHER-FIGURE'S) EDUCATION\n",
      "marout03 0.0005883792792801268 FORMAL MARITAL STATUS WHEN PREGNANCY ENDED - 3RD\n",
      "abort12 0.0005779076671296179 FA-3B RECEIVED ABORTION LAST 12 MONTHS\n",
      "p1yhsage 0.0005631662500977797 CI-6 PARTNER'S AGE AT 1ST SEX-1ST REPORTED PARTNER IN LAST 12 MOS\n",
      "numfirsm1 0.0005601782001567468 TOTAL NUMBER OF RESPONSES IN EB-1 FIRSMETH - PRESCRIPTION METHODS\n",
      "lbw1_i 0.0005499451546340239 LBW1 IMPUTATION FLAG\n",
      "agecon_i 0.0005303754891523571 AGECON IMPUTATION FLAG\n",
      "mar1con1_i 0.0005217496825273837 MAR1CON1 IMPUTATION FLAG\n",
      "nummult35 0.0005188253090530059 NUMBER OF METHODS REPORTED IN (NOV 2001)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_171984/255888651.py:29: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`\n",
      "  desc = desc[0]\n"
     ]
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "#Here are the 30 variables that yield the highest pseudo-R^2 values.\n",
    "\n",
    "MiningReport(variables)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "6d2ed94e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.691874\n",
      "         Iterations 4\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>          <td>boy</td>       <th>  No. Observations:  </th>  <td>  8884</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                 <td>Logit</td>      <th>  Df Residuals:      </th>  <td>  8880</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>  <td>     3</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Mon, 17 Jun 2024</td> <th>  Pseudo R-squ.:     </th> <td>0.001653</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>10:22:42</td>     <th>  Log-Likelihood:    </th> <td> -6146.6</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td> -6156.8</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th> <td>0.0001432</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "            <td></td>               <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>             <td>   -0.1805</td> <td>    0.118</td> <td>   -1.534</td> <td> 0.125</td> <td>   -0.411</td> <td>    0.050</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>fmarout5 == 5[T.True]</th> <td>    0.1582</td> <td>    0.049</td> <td>    3.217</td> <td> 0.001</td> <td>    0.062</td> <td>    0.255</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>infever == 1[T.True]</th>  <td>    0.2194</td> <td>    0.065</td> <td>    3.374</td> <td> 0.001</td> <td>    0.092</td> <td>    0.347</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>agepreg</th>               <td>    0.0050</td> <td>    0.004</td> <td>    1.172</td> <td> 0.241</td> <td>   -0.003</td> <td>    0.013</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/latex": [
       "\\begin{center}\n",
       "\\begin{tabular}{lclc}\n",
       "\\toprule\n",
       "\\textbf{Dep. Variable:}        &       boy        & \\textbf{  No. Observations:  } &     8884    \\\\\n",
       "\\textbf{Model:}                &      Logit       & \\textbf{  Df Residuals:      } &     8880    \\\\\n",
       "\\textbf{Method:}               &       MLE        & \\textbf{  Df Model:          } &        3    \\\\\n",
       "\\textbf{Date:}                 & Mon, 17 Jun 2024 & \\textbf{  Pseudo R-squ.:     } &  0.001653   \\\\\n",
       "\\textbf{Time:}                 &     10:22:42     & \\textbf{  Log-Likelihood:    } &   -6146.6   \\\\\n",
       "\\textbf{converged:}            &       True       & \\textbf{  LL-Null:           } &   -6156.8   \\\\\n",
       "\\textbf{Covariance Type:}      &    nonrobust     & \\textbf{  LLR p-value:       } & 0.0001432   \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "                               & \\textbf{coef} & \\textbf{std err} & \\textbf{z} & \\textbf{P$> |$z$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\textbf{Intercept}             &      -0.1805  &        0.118     &    -1.534  &         0.125        &       -0.411    &        0.050     \\\\\n",
       "\\textbf{fmarout5 == 5[T.True]} &       0.1582  &        0.049     &     3.217  &         0.001        &        0.062    &        0.255     \\\\\n",
       "\\textbf{infever == 1[T.True]}  &       0.2194  &        0.065     &     3.374  &         0.001        &        0.092    &        0.347     \\\\\n",
       "\\textbf{agepreg}               &       0.0050  &        0.004     &     1.172  &         0.241        &       -0.003    &        0.013     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "%\\caption{Logit Regression Results}\n",
       "\\end{center}"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:                    boy   No. Observations:                 8884\n",
       "Model:                          Logit   Df Residuals:                     8880\n",
       "Method:                           MLE   Df Model:                            3\n",
       "Date:                Mon, 17 Jun 2024   Pseudo R-squ.:                0.001653\n",
       "Time:                        10:22:42   Log-Likelihood:                -6146.6\n",
       "converged:                       True   LL-Null:                       -6156.8\n",
       "Covariance Type:            nonrobust   LLR p-value:                 0.0001432\n",
       "=========================================================================================\n",
       "                            coef    std err          z      P>|z|      [0.025      0.975]\n",
       "-----------------------------------------------------------------------------------------\n",
       "Intercept                -0.1805      0.118     -1.534      0.125      -0.411       0.050\n",
       "fmarout5 == 5[T.True]     0.1582      0.049      3.217      0.001       0.062       0.255\n",
       "infever == 1[T.True]      0.2194      0.065      3.374      0.001       0.092       0.347\n",
       "agepreg                   0.0050      0.004      1.172      0.241      -0.003       0.013\n",
       "=========================================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "# Eliminating variables that are not known during pregnancy and \n",
    "# others that are fishy for various reasons, here's the best model I could find:\n",
    "\n",
    "formula = 'boy ~ agepreg + fmarout5==5 + infever==1'\n",
    "model = smf.logit(formula, data=join)\n",
    "results = model.fit()\n",
    "results.summary() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b22696a2",
   "metadata": {},
   "source": [
    "**Exercise:** If the quantity you want to predict is a count, you can use Poisson regression, which is implemented in StatsModels with a function called `poisson`. It works the same way as `ols` and `logit`. As an exercise, let’s use it to predict how many children a woman has born; in the NSFG dataset, this variable is called `numbabes`.\n",
    "\n",
    "Suppose you meet a woman who is 35 years old, black, and a college graduate whose annual household income exceeds $75,000. How many children would you predict she has born?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "19bd5c7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution\n",
    "\n",
    "# I used a nonlinear model of age.\n",
    "\n",
    "join['numbabes'] = join.numbabes.replace([97], np.nan)\n",
    "join['age2'] = join.age_r**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "185c20fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 1.677002\n",
      "         Iterations 7\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Poisson Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>numbabes</td>     <th>  No. Observations:  </th>   <td>  8884</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                <td>Poisson</td>     <th>  Df Residuals:      </th>   <td>  8877</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>   <td>     6</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Mon, 17 Jun 2024</td> <th>  Pseudo R-squ.:     </th>   <td>0.03686</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>10:22:42</td>     <th>  Log-Likelihood:    </th>  <td> -14898.</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th>  <td> -15469.</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th> <td>3.681e-243</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td></td>          <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>    <td>   -1.0324</td> <td>    0.169</td> <td>   -6.098</td> <td> 0.000</td> <td>   -1.364</td> <td>   -0.701</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.2]</th> <td>   -0.1401</td> <td>    0.015</td> <td>   -9.479</td> <td> 0.000</td> <td>   -0.169</td> <td>   -0.111</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.3]</th> <td>   -0.0991</td> <td>    0.025</td> <td>   -4.029</td> <td> 0.000</td> <td>   -0.147</td> <td>   -0.051</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age_r</th>        <td>    0.1556</td> <td>    0.010</td> <td>   15.006</td> <td> 0.000</td> <td>    0.135</td> <td>    0.176</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age2</th>         <td>   -0.0020</td> <td>    0.000</td> <td>  -13.102</td> <td> 0.000</td> <td>   -0.002</td> <td>   -0.002</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>totincr</th>      <td>   -0.0187</td> <td>    0.002</td> <td>   -9.830</td> <td> 0.000</td> <td>   -0.022</td> <td>   -0.015</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>educat</th>       <td>   -0.0471</td> <td>    0.003</td> <td>  -16.076</td> <td> 0.000</td> <td>   -0.053</td> <td>   -0.041</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/latex": [
       "\\begin{center}\n",
       "\\begin{tabular}{lclc}\n",
       "\\toprule\n",
       "\\textbf{Dep. Variable:}   &     numbabes     & \\textbf{  No. Observations:  } &     8884    \\\\\n",
       "\\textbf{Model:}           &     Poisson      & \\textbf{  Df Residuals:      } &     8877    \\\\\n",
       "\\textbf{Method:}          &       MLE        & \\textbf{  Df Model:          } &        6    \\\\\n",
       "\\textbf{Date:}            & Mon, 17 Jun 2024 & \\textbf{  Pseudo R-squ.:     } &  0.03686    \\\\\n",
       "\\textbf{Time:}            &     10:22:42     & \\textbf{  Log-Likelihood:    } &   -14898.   \\\\\n",
       "\\textbf{converged:}       &       True       & \\textbf{  LL-Null:           } &   -15469.   \\\\\n",
       "\\textbf{Covariance Type:} &    nonrobust     & \\textbf{  LLR p-value:       } & 3.681e-243  \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "                      & \\textbf{coef} & \\textbf{std err} & \\textbf{z} & \\textbf{P$> |$z$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\textbf{Intercept}    &      -1.0324  &        0.169     &    -6.098  &         0.000        &       -1.364    &       -0.701     \\\\\n",
       "\\textbf{C(race)[T.2]} &      -0.1401  &        0.015     &    -9.479  &         0.000        &       -0.169    &       -0.111     \\\\\n",
       "\\textbf{C(race)[T.3]} &      -0.0991  &        0.025     &    -4.029  &         0.000        &       -0.147    &       -0.051     \\\\\n",
       "\\textbf{age\\_r}       &       0.1556  &        0.010     &    15.006  &         0.000        &        0.135    &        0.176     \\\\\n",
       "\\textbf{age2}         &      -0.0020  &        0.000     &   -13.102  &         0.000        &       -0.002    &       -0.002     \\\\\n",
       "\\textbf{totincr}      &      -0.0187  &        0.002     &    -9.830  &         0.000        &       -0.022    &       -0.015     \\\\\n",
       "\\textbf{educat}       &      -0.0471  &        0.003     &   -16.076  &         0.000        &       -0.053    &       -0.041     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "%\\caption{Poisson Regression Results}\n",
       "\\end{center}"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                          Poisson Regression Results                          \n",
       "==============================================================================\n",
       "Dep. Variable:               numbabes   No. Observations:                 8884\n",
       "Model:                        Poisson   Df Residuals:                     8877\n",
       "Method:                           MLE   Df Model:                            6\n",
       "Date:                Mon, 17 Jun 2024   Pseudo R-squ.:                 0.03686\n",
       "Time:                        10:22:42   Log-Likelihood:                -14898.\n",
       "converged:                       True   LL-Null:                       -15469.\n",
       "Covariance Type:            nonrobust   LLR p-value:                3.681e-243\n",
       "================================================================================\n",
       "                   coef    std err          z      P>|z|      [0.025      0.975]\n",
       "--------------------------------------------------------------------------------\n",
       "Intercept       -1.0324      0.169     -6.098      0.000      -1.364      -0.701\n",
       "C(race)[T.2]    -0.1401      0.015     -9.479      0.000      -0.169      -0.111\n",
       "C(race)[T.3]    -0.0991      0.025     -4.029      0.000      -0.147      -0.051\n",
       "age_r            0.1556      0.010     15.006      0.000       0.135       0.176\n",
       "age2            -0.0020      0.000    -13.102      0.000      -0.002      -0.002\n",
       "totincr         -0.0187      0.002     -9.830      0.000      -0.022      -0.015\n",
       "educat          -0.0471      0.003    -16.076      0.000      -0.053      -0.041\n",
       "================================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "formula = 'numbabes ~ age_r + age2 + age3 + C(race) + totincr + educat'\n",
    "formula = 'numbabes ~ age_r + age2 + C(race) + totincr + educat'\n",
    "model = smf.poisson(formula, data=join)\n",
    "results = model.fit()\n",
    "results.summary() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e39e2e87",
   "metadata": {},
   "source": [
    "Now we can predict the number of children for a woman who is 35 years old, black, and a college\n",
    "graduate whose annual household income exceeds $75,000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "c593fb4a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2.496802\n",
       "dtype: float64"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "columns = ['age_r', 'age2', 'age3', 'race', 'totincr', 'educat']\n",
    "new = pd.DataFrame([[35, 35**2, 35**3, 1, 14, 16]], columns=columns)\n",
    "results.predict(new)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e34d2bb",
   "metadata": {},
   "source": [
    "**Exercise:** If the quantity you want to predict is categorical, you can use multinomial logistic regression, which is implemented in StatsModels with a function called `mnlogit`. As an exercise, let’s use it to guess whether a woman is married, cohabitating, widowed, divorced, separated, or never married; in the NSFG dataset, marital status is encoded in a variable called `rmarital`.\n",
    "\n",
    "Suppose you meet a woman who is 25 years old, white, and a high school graduate whose annual household income is about $45,000. What is the probability that she is married, cohabitating, etc?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "77e41e64",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 1.084053\n",
      "         Iterations 8\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>MNLogit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>rmarital</td>     <th>  No. Observations:  </th>  <td>  8884</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                <td>MNLogit</td>     <th>  Df Residuals:      </th>  <td>  8849</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>  <td>    30</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Mon, 17 Jun 2024</td> <th>  Pseudo R-squ.:     </th>  <td>0.1682</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>10:22:43</td>     <th>  Log-Likelihood:    </th> <td> -9630.7</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td> -11579.</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th>  <td> 0.000</td> \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "   <th>rmarital=2</th>     <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>    <td>    9.0156</td> <td>    0.805</td> <td>   11.199</td> <td> 0.000</td> <td>    7.438</td> <td>   10.593</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.2]</th> <td>   -0.9237</td> <td>    0.089</td> <td>  -10.418</td> <td> 0.000</td> <td>   -1.097</td> <td>   -0.750</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.3]</th> <td>   -0.6179</td> <td>    0.136</td> <td>   -4.536</td> <td> 0.000</td> <td>   -0.885</td> <td>   -0.351</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age_r</th>        <td>   -0.3635</td> <td>    0.051</td> <td>   -7.150</td> <td> 0.000</td> <td>   -0.463</td> <td>   -0.264</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age2</th>         <td>    0.0048</td> <td>    0.001</td> <td>    6.103</td> <td> 0.000</td> <td>    0.003</td> <td>    0.006</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>totincr</th>      <td>   -0.1310</td> <td>    0.012</td> <td>  -11.337</td> <td> 0.000</td> <td>   -0.154</td> <td>   -0.108</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>educat</th>       <td>   -0.1953</td> <td>    0.019</td> <td>  -10.424</td> <td> 0.000</td> <td>   -0.232</td> <td>   -0.159</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <th>rmarital=3</th>     <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>    <td>    2.9570</td> <td>    3.020</td> <td>    0.979</td> <td> 0.328</td> <td>   -2.963</td> <td>    8.877</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.2]</th> <td>   -0.4411</td> <td>    0.237</td> <td>   -1.863</td> <td> 0.062</td> <td>   -0.905</td> <td>    0.023</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.3]</th> <td>    0.0591</td> <td>    0.336</td> <td>    0.176</td> <td> 0.860</td> <td>   -0.600</td> <td>    0.718</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age_r</th>        <td>   -0.3177</td> <td>    0.177</td> <td>   -1.798</td> <td> 0.072</td> <td>   -0.664</td> <td>    0.029</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age2</th>         <td>    0.0064</td> <td>    0.003</td> <td>    2.528</td> <td> 0.011</td> <td>    0.001</td> <td>    0.011</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>totincr</th>      <td>   -0.3258</td> <td>    0.032</td> <td>  -10.175</td> <td> 0.000</td> <td>   -0.389</td> <td>   -0.263</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>educat</th>       <td>   -0.0991</td> <td>    0.048</td> <td>   -2.050</td> <td> 0.040</td> <td>   -0.194</td> <td>   -0.004</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <th>rmarital=4</th>     <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>    <td>   -3.5238</td> <td>    1.205</td> <td>   -2.924</td> <td> 0.003</td> <td>   -5.886</td> <td>   -1.162</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.2]</th> <td>   -0.3213</td> <td>    0.093</td> <td>   -3.445</td> <td> 0.001</td> <td>   -0.504</td> <td>   -0.139</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.3]</th> <td>   -0.7706</td> <td>    0.171</td> <td>   -4.509</td> <td> 0.000</td> <td>   -1.106</td> <td>   -0.436</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age_r</th>        <td>    0.1155</td> <td>    0.071</td> <td>    1.626</td> <td> 0.104</td> <td>   -0.024</td> <td>    0.255</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age2</th>         <td>   -0.0007</td> <td>    0.001</td> <td>   -0.701</td> <td> 0.483</td> <td>   -0.003</td> <td>    0.001</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>totincr</th>      <td>   -0.2276</td> <td>    0.012</td> <td>  -19.621</td> <td> 0.000</td> <td>   -0.250</td> <td>   -0.205</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>educat</th>       <td>    0.0667</td> <td>    0.017</td> <td>    3.995</td> <td> 0.000</td> <td>    0.034</td> <td>    0.099</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <th>rmarital=5</th>     <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>    <td>   -2.8963</td> <td>    1.305</td> <td>   -2.220</td> <td> 0.026</td> <td>   -5.453</td> <td>   -0.339</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.2]</th> <td>   -1.0407</td> <td>    0.104</td> <td>  -10.038</td> <td> 0.000</td> <td>   -1.244</td> <td>   -0.837</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.3]</th> <td>   -0.5661</td> <td>    0.156</td> <td>   -3.635</td> <td> 0.000</td> <td>   -0.871</td> <td>   -0.261</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age_r</th>        <td>    0.2411</td> <td>    0.079</td> <td>    3.038</td> <td> 0.002</td> <td>    0.086</td> <td>    0.397</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age2</th>         <td>   -0.0035</td> <td>    0.001</td> <td>   -2.977</td> <td> 0.003</td> <td>   -0.006</td> <td>   -0.001</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>totincr</th>      <td>   -0.2932</td> <td>    0.015</td> <td>  -20.159</td> <td> 0.000</td> <td>   -0.322</td> <td>   -0.265</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>educat</th>       <td>   -0.0174</td> <td>    0.021</td> <td>   -0.813</td> <td> 0.416</td> <td>   -0.059</td> <td>    0.025</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <th>rmarital=6</th>     <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>    <td>    8.0533</td> <td>    0.814</td> <td>    9.890</td> <td> 0.000</td> <td>    6.457</td> <td>    9.649</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.2]</th> <td>   -2.1871</td> <td>    0.080</td> <td>  -27.211</td> <td> 0.000</td> <td>   -2.345</td> <td>   -2.030</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>C(race)[T.3]</th> <td>   -1.9611</td> <td>    0.138</td> <td>  -14.188</td> <td> 0.000</td> <td>   -2.232</td> <td>   -1.690</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age_r</th>        <td>   -0.2127</td> <td>    0.052</td> <td>   -4.122</td> <td> 0.000</td> <td>   -0.314</td> <td>   -0.112</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age2</th>         <td>    0.0019</td> <td>    0.001</td> <td>    2.321</td> <td> 0.020</td> <td>    0.000</td> <td>    0.003</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>totincr</th>      <td>   -0.2945</td> <td>    0.012</td> <td>  -25.320</td> <td> 0.000</td> <td>   -0.317</td> <td>   -0.272</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>educat</th>       <td>   -0.0742</td> <td>    0.018</td> <td>   -4.169</td> <td> 0.000</td> <td>   -0.109</td> <td>   -0.039</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/latex": [
       "\\begin{center}\n",
       "\\begin{tabular}{lclc}\n",
       "\\toprule\n",
       "\\textbf{Dep. Variable:}   &     rmarital     & \\textbf{  No. Observations:  } &     8884    \\\\\n",
       "\\textbf{Model:}           &     MNLogit      & \\textbf{  Df Residuals:      } &     8849    \\\\\n",
       "\\textbf{Method:}          &       MLE        & \\textbf{  Df Model:          } &       30    \\\\\n",
       "\\textbf{Date:}            & Mon, 17 Jun 2024 & \\textbf{  Pseudo R-squ.:     } &   0.1682    \\\\\n",
       "\\textbf{Time:}            &     10:22:43     & \\textbf{  Log-Likelihood:    } &   -9630.7   \\\\\n",
       "\\textbf{converged:}       &       True       & \\textbf{  LL-Null:           } &   -11579.   \\\\\n",
       "\\textbf{Covariance Type:} &    nonrobust     & \\textbf{  LLR p-value:       } &    0.000    \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{ccccccc}\n",
       " \\textbf{rmarital=2}  & \\textbf{coef} & \\textbf{std err} & \\textbf{z} & \\textbf{P$> |$z$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "\\textbf{Intercept}    &       9.0156  &        0.805     &    11.199  &         0.000        &        7.438    &       10.593     \\\\\n",
       "\\textbf{C(race)[T.2]} &      -0.9237  &        0.089     &   -10.418  &         0.000        &       -1.097    &       -0.750     \\\\\n",
       "\\textbf{C(race)[T.3]} &      -0.6179  &        0.136     &    -4.536  &         0.000        &       -0.885    &       -0.351     \\\\\n",
       "\\textbf{age\\_r}       &      -0.3635  &        0.051     &    -7.150  &         0.000        &       -0.463    &       -0.264     \\\\\n",
       "\\textbf{age2}         &       0.0048  &        0.001     &     6.103  &         0.000        &        0.003    &        0.006     \\\\\n",
       "\\textbf{totincr}      &      -0.1310  &        0.012     &   -11.337  &         0.000        &       -0.154    &       -0.108     \\\\\n",
       "\\textbf{educat}       &      -0.1953  &        0.019     &   -10.424  &         0.000        &       -0.232    &       -0.159     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{ccccccc}\n",
       " \\textbf{rmarital=3}  & \\textbf{coef} & \\textbf{std err} & \\textbf{z} & \\textbf{P$> |$z$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "\\textbf{Intercept}    &       2.9570  &        3.020     &     0.979  &         0.328        &       -2.963    &        8.877     \\\\\n",
       "\\textbf{C(race)[T.2]} &      -0.4411  &        0.237     &    -1.863  &         0.062        &       -0.905    &        0.023     \\\\\n",
       "\\textbf{C(race)[T.3]} &       0.0591  &        0.336     &     0.176  &         0.860        &       -0.600    &        0.718     \\\\\n",
       "\\textbf{age\\_r}       &      -0.3177  &        0.177     &    -1.798  &         0.072        &       -0.664    &        0.029     \\\\\n",
       "\\textbf{age2}         &       0.0064  &        0.003     &     2.528  &         0.011        &        0.001    &        0.011     \\\\\n",
       "\\textbf{totincr}      &      -0.3258  &        0.032     &   -10.175  &         0.000        &       -0.389    &       -0.263     \\\\\n",
       "\\textbf{educat}       &      -0.0991  &        0.048     &    -2.050  &         0.040        &       -0.194    &       -0.004     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{ccccccc}\n",
       " \\textbf{rmarital=4}  & \\textbf{coef} & \\textbf{std err} & \\textbf{z} & \\textbf{P$> |$z$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "\\textbf{Intercept}    &      -3.5238  &        1.205     &    -2.924  &         0.003        &       -5.886    &       -1.162     \\\\\n",
       "\\textbf{C(race)[T.2]} &      -0.3213  &        0.093     &    -3.445  &         0.001        &       -0.504    &       -0.139     \\\\\n",
       "\\textbf{C(race)[T.3]} &      -0.7706  &        0.171     &    -4.509  &         0.000        &       -1.106    &       -0.436     \\\\\n",
       "\\textbf{age\\_r}       &       0.1155  &        0.071     &     1.626  &         0.104        &       -0.024    &        0.255     \\\\\n",
       "\\textbf{age2}         &      -0.0007  &        0.001     &    -0.701  &         0.483        &       -0.003    &        0.001     \\\\\n",
       "\\textbf{totincr}      &      -0.2276  &        0.012     &   -19.621  &         0.000        &       -0.250    &       -0.205     \\\\\n",
       "\\textbf{educat}       &       0.0667  &        0.017     &     3.995  &         0.000        &        0.034    &        0.099     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{ccccccc}\n",
       " \\textbf{rmarital=5}  & \\textbf{coef} & \\textbf{std err} & \\textbf{z} & \\textbf{P$> |$z$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "\\textbf{Intercept}    &      -2.8963  &        1.305     &    -2.220  &         0.026        &       -5.453    &       -0.339     \\\\\n",
       "\\textbf{C(race)[T.2]} &      -1.0407  &        0.104     &   -10.038  &         0.000        &       -1.244    &       -0.837     \\\\\n",
       "\\textbf{C(race)[T.3]} &      -0.5661  &        0.156     &    -3.635  &         0.000        &       -0.871    &       -0.261     \\\\\n",
       "\\textbf{age\\_r}       &       0.2411  &        0.079     &     3.038  &         0.002        &        0.086    &        0.397     \\\\\n",
       "\\textbf{age2}         &      -0.0035  &        0.001     &    -2.977  &         0.003        &       -0.006    &       -0.001     \\\\\n",
       "\\textbf{totincr}      &      -0.2932  &        0.015     &   -20.159  &         0.000        &       -0.322    &       -0.265     \\\\\n",
       "\\textbf{educat}       &      -0.0174  &        0.021     &    -0.813  &         0.416        &       -0.059    &        0.025     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{ccccccc}\n",
       " \\textbf{rmarital=6}  & \\textbf{coef} & \\textbf{std err} & \\textbf{z} & \\textbf{P$> |$z$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "\\textbf{Intercept}    &       8.0533  &        0.814     &     9.890  &         0.000        &        6.457    &        9.649     \\\\\n",
       "\\textbf{C(race)[T.2]} &      -2.1871  &        0.080     &   -27.211  &         0.000        &       -2.345    &       -2.030     \\\\\n",
       "\\textbf{C(race)[T.3]} &      -1.9611  &        0.138     &   -14.188  &         0.000        &       -2.232    &       -1.690     \\\\\n",
       "\\textbf{age\\_r}       &      -0.2127  &        0.052     &    -4.122  &         0.000        &       -0.314    &       -0.112     \\\\\n",
       "\\textbf{age2}         &       0.0019  &        0.001     &     2.321  &         0.020        &        0.000    &        0.003     \\\\\n",
       "\\textbf{totincr}      &      -0.2945  &        0.012     &   -25.320  &         0.000        &       -0.317    &       -0.272     \\\\\n",
       "\\textbf{educat}       &      -0.0742  &        0.018     &    -4.169  &         0.000        &       -0.109    &       -0.039     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "%\\caption{MNLogit Regression Results}\n",
       "\\end{center}"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                          MNLogit Regression Results                          \n",
       "==============================================================================\n",
       "Dep. Variable:               rmarital   No. Observations:                 8884\n",
       "Model:                        MNLogit   Df Residuals:                     8849\n",
       "Method:                           MLE   Df Model:                           30\n",
       "Date:                Mon, 17 Jun 2024   Pseudo R-squ.:                  0.1682\n",
       "Time:                        10:22:43   Log-Likelihood:                -9630.7\n",
       "converged:                       True   LL-Null:                       -11579.\n",
       "Covariance Type:            nonrobust   LLR p-value:                     0.000\n",
       "================================================================================\n",
       "  rmarital=2       coef    std err          z      P>|z|      [0.025      0.975]\n",
       "--------------------------------------------------------------------------------\n",
       "Intercept        9.0156      0.805     11.199      0.000       7.438      10.593\n",
       "C(race)[T.2]    -0.9237      0.089    -10.418      0.000      -1.097      -0.750\n",
       "C(race)[T.3]    -0.6179      0.136     -4.536      0.000      -0.885      -0.351\n",
       "age_r           -0.3635      0.051     -7.150      0.000      -0.463      -0.264\n",
       "age2             0.0048      0.001      6.103      0.000       0.003       0.006\n",
       "totincr         -0.1310      0.012    -11.337      0.000      -0.154      -0.108\n",
       "educat          -0.1953      0.019    -10.424      0.000      -0.232      -0.159\n",
       "--------------------------------------------------------------------------------\n",
       "  rmarital=3       coef    std err          z      P>|z|      [0.025      0.975]\n",
       "--------------------------------------------------------------------------------\n",
       "Intercept        2.9570      3.020      0.979      0.328      -2.963       8.877\n",
       "C(race)[T.2]    -0.4411      0.237     -1.863      0.062      -0.905       0.023\n",
       "C(race)[T.3]     0.0591      0.336      0.176      0.860      -0.600       0.718\n",
       "age_r           -0.3177      0.177     -1.798      0.072      -0.664       0.029\n",
       "age2             0.0064      0.003      2.528      0.011       0.001       0.011\n",
       "totincr         -0.3258      0.032    -10.175      0.000      -0.389      -0.263\n",
       "educat          -0.0991      0.048     -2.050      0.040      -0.194      -0.004\n",
       "--------------------------------------------------------------------------------\n",
       "  rmarital=4       coef    std err          z      P>|z|      [0.025      0.975]\n",
       "--------------------------------------------------------------------------------\n",
       "Intercept       -3.5238      1.205     -2.924      0.003      -5.886      -1.162\n",
       "C(race)[T.2]    -0.3213      0.093     -3.445      0.001      -0.504      -0.139\n",
       "C(race)[T.3]    -0.7706      0.171     -4.509      0.000      -1.106      -0.436\n",
       "age_r            0.1155      0.071      1.626      0.104      -0.024       0.255\n",
       "age2            -0.0007      0.001     -0.701      0.483      -0.003       0.001\n",
       "totincr         -0.2276      0.012    -19.621      0.000      -0.250      -0.205\n",
       "educat           0.0667      0.017      3.995      0.000       0.034       0.099\n",
       "--------------------------------------------------------------------------------\n",
       "  rmarital=5       coef    std err          z      P>|z|      [0.025      0.975]\n",
       "--------------------------------------------------------------------------------\n",
       "Intercept       -2.8963      1.305     -2.220      0.026      -5.453      -0.339\n",
       "C(race)[T.2]    -1.0407      0.104    -10.038      0.000      -1.244      -0.837\n",
       "C(race)[T.3]    -0.5661      0.156     -3.635      0.000      -0.871      -0.261\n",
       "age_r            0.2411      0.079      3.038      0.002       0.086       0.397\n",
       "age2            -0.0035      0.001     -2.977      0.003      -0.006      -0.001\n",
       "totincr         -0.2932      0.015    -20.159      0.000      -0.322      -0.265\n",
       "educat          -0.0174      0.021     -0.813      0.416      -0.059       0.025\n",
       "--------------------------------------------------------------------------------\n",
       "  rmarital=6       coef    std err          z      P>|z|      [0.025      0.975]\n",
       "--------------------------------------------------------------------------------\n",
       "Intercept        8.0533      0.814      9.890      0.000       6.457       9.649\n",
       "C(race)[T.2]    -2.1871      0.080    -27.211      0.000      -2.345      -2.030\n",
       "C(race)[T.3]    -1.9611      0.138    -14.188      0.000      -2.232      -1.690\n",
       "age_r           -0.2127      0.052     -4.122      0.000      -0.314      -0.112\n",
       "age2             0.0019      0.001      2.321      0.020       0.000       0.003\n",
       "totincr         -0.2945      0.012    -25.320      0.000      -0.317      -0.272\n",
       "educat          -0.0742      0.018     -4.169      0.000      -0.109      -0.039\n",
       "================================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "# Here's the best model I could find.\n",
    "\n",
    "formula='rmarital ~ age_r + age2 + C(race) + totincr + educat'\n",
    "model = smf.mnlogit(formula, data=join)\n",
    "results = model.fit()\n",
    "results.summary() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5bb8163",
   "metadata": {},
   "source": [
    "Make a prediction for a woman who is 25 years old, white, and a high\n",
    "school graduate whose annual household income is about $45,000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "4bb62cc7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.750028</td>\n",
       "      <td>0.126397</td>\n",
       "      <td>0.001564</td>\n",
       "      <td>0.033403</td>\n",
       "      <td>0.021485</td>\n",
       "      <td>0.067122</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1         2         3         4         5\n",
       "0  0.750028  0.126397  0.001564  0.033403  0.021485  0.067122"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "# This person has a 75% chance of being currently married, \n",
    "# a 13% chance of being \"not married but living with opposite \n",
    "# sex partner\", etc.\n",
    "\n",
    "columns = ['age_r', 'age2', 'race', 'totincr', 'educat']\n",
    "new = pd.DataFrame([[25, 25**2, 2, 11, 12]], columns=columns)\n",
    "results.predict(new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23da96b4",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
